Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Help me make (statistical) sense of these DTS test results (Read 19976 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Help me make (statistical) sense of these DTS test results

Being often confronted with claims that DTS and especially Dolby 5.1 sounds like 'crap' compared to lossless surround -- to the extent that some simply refuse to buy a surround remix if it isn't offered in lossless --  I often  find myself arguing that DTS and DD are actually rather good at what they do (lossy perceptual encoding), and should be hard to distinguish from lossless in typical listening.  Suffice to say I am sometimes met with vigorous, occasionally bordering on vicious, reactions to this stance 

That happened on the SurroundSound forum a few months ago, and in response, a poster (a nice guy, not vicious!) set up a listening test.  He took a lossless 96/24 surround track of subjectively good audio quality, and converted it to DTS with Surcode's encoder, at three different settings.  He then converted all of the lossy versions to 96/24 PCM to match the lossless, and offered them as a downloadable package that one could use to compare 'blind' -- the subject doesn't know what order the 4 different versions are in (though he does know what formats were used in the test) , and is tasked with identifying them. 

Nine people so far have returned answers  (I haven't yet, due to pesky amounts of travel I've been doing lately) and the author has kindlyl shared these preliminary results.  Red indicates incorrect identifications





With the caveat that this is hardly a rigorous scientific test (which the test author has acknowledged all along) , and assuming good faith on the part of the subjects, I find these results surprising.  I'm also at a loss to evaluate the probabilities here. Not just the fact that it's a four-way choice , but also because some subjects only got as far as ID ing the lossless.  As a statistics maven friend of mine said, "We have to either look at a test of 9 people on identifying the original versus other, or a test of 7 people on picking all four versions."  He also tells me the number of replies here is too small to do a chi-square test to determine p-value the traditional way.  And also "I think there is evidence here (presuming the experiment was carried out appropriately) that people can identify the original audio (6/9 better than 25%), and that they can identify all four versions (2/7 better than 4%)."  I agreed that it's possible -- no one has ever denied that -- but I was surprised at how *many* were able to do it here.  So he and I  explored the significance of the number of correct lossless vs not lossless  replies (6/9) .  My stats maven's analysis of that:

Quote
The probability of 6 guessing correctly out of 9 is...

1/4 * 1/4 * 1/4 * 1/4 * 1/4 * 1/4 * 3/4 * 3/4 * 3/4

(i.e. 6 people do something with a 1 in 4 chance and 3 people do the opposite, i.e. something with a 3 in 4 chance)

... but we then have to multiply that by what's called 9C6, that is how many ways there are of picking 6 people from 9 people, because we don't care which 6 people are right. 6C9 is the same as 9C3, i.e. the number of ways of picking 3 people from 9 people = 9!/(3!*6!) = 9*8*7/3*2

So, the whole thing comes to 0.008652. Pretty unlikely.

That's not a p-value however. The p-value would be slightly larger because it would be the probability of 6 OR MORE people guessing correctly, not of PRECISELY 6 guessing correctly, which is the number above.

So, we calculate the probability of 7 guessing correctly...
0.001236

And of 8...
0.000103

And of 9...
3.8147E-06

So, summing those...

The probability that 6 or more people would guess correctly with a 1/4 chance is 0.001, so that's our p-value. Highly significant


i.e., highly significant assuming a significance threshold of p=0.05 (which we agreed is traditional but not always appropriate, sometimes it needs to be smaller, e.g. if there's good independent  reason to believe the phenomenon should be very unlikely or even 'impossible'). 

NB This wasn't performed as an ABC/hr type preference test, as would normally be done with codecs. It's just people listening and comparing at home, by various (unspecified) means.  (I'm leaving out lots of back and forth from various forum members, one of whom for example  pointed up a section of music that he felt especially reveals the differences) 


I'd be curious to hear what HA has to say.  I can probably even provide the original sound files if people want to try it themselves (the order shown in the table is not necessarily the file order).

Help me make (statistical) sense of these DTS test results

Reply #1
I can probably even provide the original sound files if people want to try it themselves
Who can resist that?

A link to the original discussion would be interesting too.

FWIW I don't think there's any general belief that 755kbps DTS is transparent. Ditto 384kbps AC-3 (not tested here).

Cheers,
David.


Help me make (statistical) sense of these DTS test results

Reply #2
Quote
Being often confronted with claims that DTS and especially Dolby 5.1 sounds like 'crap' compared to lossless surround
I certainly wouldn't say they sound like crap! 

I've never heard a lossless surround file, but I have a shelf-full of concert DVDs, most with Dolby or DTS surround.  Some of the BEST sound I've ever heard comes from these DVDs.  When the DVD offeres an "apples-to-oranges" choice between lossless LPCM stereo and lossy surround, I'll always choose the surround track 

And, I'd say my "best-sounding" 5.1 DVDs sound better than my "best sounding" CDs.  Again, apples-to-oranges with totally different recordings on the DVDs & CDs.    (I've also got a few lousy-sounding 5.1 DVD concerts.)

So, maybe I'm a sucker for surround...    And since I'll choose lossy 5.1 over lossless stereo, it doesn't sound like crap to me!


Help me make (statistical) sense of these DTS test results

Reply #3
I can probably even provide the original sound files if people want to try it themselves
Who can resist that?

A link to the original discussion would be interesting too.


Of course -- here you go:

here's the thread specifically for the test
https://groups.google.com/group/surroundsou...86c0bcccd987acc

the first post has a link to the audio file archive.


the impetus for the test came from this thread, apparently from my post concerning what I 'expect' to be the case when comparing DTS  to source (in this case, a DTS 96/24 version offered on an ELO remaster):
https://groups.google.com/group/surroundsou...5578d81e32cfc75


and the animosity towards me in that thread starts in this thread...which I'm sure will not surprise you once you see its subject   
https://groups.google.com/group/surroundsou...25cd151b373e63f




Quote
FWIW I don't think there's any general belief that 755kbps DTS is transparent. Ditto 384kbps AC-3 (not tested here).



Transparent under all conditions, certainly not -- but can we expect it to be easily, instantly distinguishable from source, and as a quality degradation? That is the sort of ability audiophiles claim -- that they could tell 'right away' that it was DTS and not lossless.  Along with the dismissive 'even my spouse could hear it' sort of thing, the 'ugh, I won't buy it if its only DTS' thing.    There's some of that on the ELO thread above.  Also, you can see that 755 kbps was one of three DTS versions tested --  the other two were at the 2X higher bitrate, one also being DTS 96/24 -- these results are suggesting  that 6 out 9 listeners could tell the lossless original from among a set containing a 1550 kbps DTS 96/24 version.  That seems extraordinary to me.

Help me make (statistical) sense of these DTS test results

Reply #4
Quote
Being often confronted with claims that DTS and especially Dolby 5.1 sounds like 'crap' compared to lossless surround
I certainly wouldn't say they sound like crap! 

I've never heard a lossless surround file, but I have a shelf-full of concert DVDs, most with Dolby or DTS surround.  Some of the BEST sound I've ever heard comes from these DVDs.  When the DVD offeres an "apples-to-oranges" choice between lossless LPCM stereo and lossy surround, I'll always choose the surround track 

And, I'd say my "best-sounding" 5.1 DVDs sound better than my "best sounding" CDs.  Again, apples-to-oranges with totally different recordings on the DVDs & CDs.    (I've also got a few lousy-sounding 5.1 DVD concerts.)

So, maybe I'm a sucker for surround...    And since I'll choose lossy 5.1 over lossless stereo, it doesn't sound like crap to me!


The comparison is specifically between surround versions. It is not a debate between 2-ch vs multichannel.  (Though for me, it would be easier to use ABX to compare 2-channel versions of anything).


Help me make (statistical) sense of these DTS test results

Reply #6
Wow - a DVD audio disc image. That's real handy for an ABX test!

Don't worry, I'll try it though. I knew my DVD-A player would come in useful for something  other than playing Bob the Builder DVDs of course.

Cheers,
David.

Help me make (statistical) sense of these DTS test results

Reply #7
I can only listen in stereo, and I cannot hear any difference between the four selections in the stereo mixdown (as presented by fb2k as the second track in the AOB).

However, none of these stereo mixdowns are 2496 lossless tracks. They're all devoid of real content above 24kHz. Two have what I would judge to be distortion above 24kHz, but it's not real content. There's a sharp filter at 24kHz.

I might try to listen to pair of channels from the 5.1 mix, in stereo. Though that's kind of cheating, and I don't think I have any software that will let me do this anyway.

Cheers,
David.

Help me make (statistical) sense of these DTS test results

Reply #8
btw, the test author has also since offered up a test of hi rz vs redbook.....
https://groups.google.com/group/surroundsou...a5ca293f8c20d24
This is a nice recording, though obviously SACD/DSD sourced (rising noise above 50kHz) - but one of the stereo 48kHz conversions is messed up - there's aliasing from 24kHz-48kHz. The other one is fine (no aliasing at all).

lossyWAV can only find a few brief moments where it thinks the track needs more than 16-bits. It's happy with 14-bit for the vast majority and 12 for more than half of it. (without noise shaping).

I can't check it properly because I can't play 192kHz - but at 96kHz I can't hear anything wrong with any of them, even during the very quietest parts.

FWIW I suspect some use of dynamic range compression in the original recording, but it sounds fine to me.

Cheers,
David.

Help me make (statistical) sense of these DTS test results

Reply #9
Wow - a DVD audio disc image. That's real handy for an ABX test!



    I've noticed that both QuadraphonicQuad and SurroundSound forum users are still mostly of a  'hardware player' mindset. Lots of discussion over which Oppo player plays which formats etc.


That said, I'm not even sure foobar2k's ABX feature allows ABX comparison of multichannel files...anyone know?


Help me make (statistical) sense of these DTS test results

Reply #10
I can only listen in stereo, and I cannot hear any difference between the four selections in the stereo mixdown (as presented by fb2k as the second track in the AOB).

However, none of these stereo mixdowns are 2496 lossless tracks. They're all devoid of real content above 24kHz. Two have what I would judge to be distortion above 24kHz, but it's not real content. There's a sharp filter at 24kHz.


OK, suggests the lossless recording went through a 48kHz SR step at some point.  I can ask grill (the test author) to comment on the track specs and conversion workflow.  Your mixdown software doesn't resample at 48kHz, does it?

(I purposely haven't tried to analyze these objectively, except to take rough -- 'by eye' -- peak  SPL meter readings)

Quote
I might try to listen to pair of channels from the 5.1 mix, in stereo. Though that's kind of cheating, and I don't think I have any software that will let me do this anyway.

Cheers,
David.



I played them all in 5.1 on my main system a few times, and couldn't make any confident identifications.

One of the SurroundSound listeners suggested focusing on  "section 2.20 to 2.45 on each and listen for decay, smear and compression" .  I'm going to try that when i have time.  (He also reported "The interesting thing is that the DVD-A didn't half try hard to mask the evident smear, bordering on drop-outs on two of the tracks (we can guess which these are) which on my study PC was almost unlistenable to me. ")

Help me make (statistical) sense of these DTS test results

Reply #11
btw my facts about the codec were incorrect -- according to grill's first post on the test thread,

Quote
The hotlinked Hotfile link below
leads you to a DVD-audio disc image that contains four 5.1 24/96
tracks of the same audio material encoded as:
- unaltered original MLP 5.1 24/96,
- DTS 5.1 (755 kbps),
- DTS 5.1 (1510 kbps),
- DTS 24/96 5.1 (1510 kbps).

DTS-HD MAS 2.50.20 was used for DTS encoding and AudioMuxer with
ArcSoft DTS decoder
was used for DTS decoding.



NB (obviously) that the audio was lossy encoded and then decoded, in order to generate the final lossy MLP files.


Help me make (statistical) sense of these DTS test results

Reply #13
OK, suggests the lossless recording went through a 48kHz SR step at some point.  I can ask grill (the test author) to comment on the track specs and conversion workflow.  Your mixdown software doesn't resample at 48kHz, does it?
It's the stereo mixdown, revealed in fb2k as a separate track from the 5.1 version, which I'm guessing is the stereo mixdown "automatically"(?) generated using coefficients in the MLP stream (i.e. not really a separate track on disc). It's certainly coming out as a 96kHz wave file.

OK, I've checked, and on the stereo mixdown, two of the four versions are identical. I mean absolutely bit-identical.

Either this stereo mixdown is something weird/different, or there's a simple mistake.

Cheers,
David.

 

Help me make (statistical) sense of these DTS test results

Reply #14
OK, suggests the lossless recording went through a 48kHz SR step at some point.  I can ask grill (the test author) to comment on the track specs and conversion workflow.  Your mixdown software doesn't resample at 48kHz, does it?
It's the stereo mixdown, revealed in fb2k as a separate track from the 5.1 version, which I'm guessing is the stereo mixdown "automatically"(?) generated using coefficients in the MLP stream (i.e. not really a separate track on disc). It's certainly coming out as a 96kHz wave file.


How does one get f2k to show that separate track information? DVDAExplorer shows me just 4 tracks in the mounted .iso image.


Quote
OK, I've checked, and on the stereo mixdown, two of the four versions are identical. I mean absolutely bit-identical.

Either this stereo mixdown is something weird/different, or there's a simple mistake.

Cheers,
David.


That's odd.  To be clear the downmix of which you write, is that of just the putative 'original' or are you getting 'automatic' downmixes of all four versions?  (From your phrasing I presume the latter) 

Does MLP encoding normally encode specific instruction for a 2ch mixdown (i know this *can* be done, and was done on some early DVD-As that had no remastered 2-channel track), or is that something your software is doing blindly?

Btw, the author verifies that the original audio is nominally 96/24 , but in fact has no content above 24kHz (as is rather too common among commercial 'high rez' releases).

Help me make (statistical) sense of these DTS test results

Reply #15
OK, I've checked, and on the stereo mixdown, two of the four versions are identical. I mean absolutely bit-identical.

Either this stereo mixdown is something weird/different, or there's a simple mistake.
The simple mistake was mine. Sorry. I must have compared the wrong parts.



I'm not sure where the stereo mixdown is driven from. I installed the top one of these...
http://sourceforge.net/projects/dvdadecoder/files/

What I've just discovered is that, if you drop Find_the_original.iso into fb2k, you get four separate 5.1 tracks.

Whereas if you drop the ATS_01_1.AOB from inside the ISO (7-zip will let you look inside), you get two tracks - one is a concatenation of all four 5.1 tracks, the other is the same, but with "(stereo downmix)" after it.


I'm guessing it's a fb2k plugin thing, as it tries to make sense of the ATS_01_1.AOB file without the track info in the other files in AUDIO_TS.

I'm out of my depth here - I don't have anything that will open the six channel .wav files created by decoding the 5.1 mixes, so I can do no more.

I'm not sure why anyone would want to buy this content sampled at 96kHz. It was obviously upsampled from 48kHz. The original master was 48kHz. The best thing you could buy would be the original master!

Cheers,
David.

Help me make (statistical) sense of these DTS test results

Reply #16

Hi, you can probably mount the ISO in a virtual drive and then extract the .wav files with DVDAExplorer. If you are using a Windows OS that is, for a Mac I wouldn't know...

Help me make (statistical) sense of these DTS test results

Reply #17
Hi, you can probably mount the ISO in a virtual drive and then extract the .wav files with DVDAExplorer. If you are using a Windows OS that is, for a Mac I wouldn't know...


Yeah, that's not a problem.  2bdecided, what is it you wanted to do with the six-channel wavs?