Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe (Read 172893 times) previous topic - next topic
0 Members and 2 Guests are viewing this topic.

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

"Audibility of a CD-Standard A/D/A Loop Inserted into High-Resolution Audio Playback". E. Brad Meyer and David R. Moran. JAES 55(9) September 2007. It's worth noting that members of the BAS wrote the paper and performed in the tests. You'll need an AES membership to access the article, so no link.

Abstract:
Quote
Claims both published and anecdotal are regularly made for audibly superior sound quality for two-channel audio encoded with longer word lengths and/or at higher sampling rates than the 16-bit/44.1-kHz CD standard. The authors report on a series of double-blind tests comparing the analog output of high-resolution players playing high-resolution recordings with the same signal passed through a 16-bit/44.1-kHz "bottleneck." The tests were conducted for over a year using different systems and a variety of subjects. The systems included expensive professional monitors and one high-end system with electrostatic loudspeakers and expensive components and cables. The subjects included professional recording engineers, students in a university recording program, and dedicated audiophiles. The test results show that the CD-quality A/D/A loop was undetectable at normal-to-loud listening levels, by any of the subjects, on any of the playback systems. The noise of the CD-quality loop was audible only at very elevated levels.

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #1
Interesting, thanks!
Over thinking, over analyzing separates the body from the mind.

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #2
Very interesting...people are still going to claim they hear a difference anyway, though...

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #3
Thanks for the info. Any details about the A/D/A loop ?
Discussing a publication based only on the abstract can be quite dangerous. However, complete AES articles are only available to members, or can be bought by non-members for $ 20 each. It is not allowed to share copies of AES documents with others. This makes public discussion rather difficult, although "It is permitted to quote from this Journal with customary credit to the source."
Interesting papers like this make me consider renewing my AES membership to have access to the online JAES and their On-line Electronic Library, despite the restrictions

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #4
Thanks for this
EAC>1)fb2k>LAME3.99 -V 0 --vbr-new>WMP12 2)MAC-Extra High

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #5
The test results show that the CD-quality A/D/A loop was undetectable at normal-to-loud listening levels, by any of the subjects, on any of the playback systems. The noise of the CD-quality loop was audible only at very elevated levels.

Sorry, does this mean they were testing whether this noise from the ADA conversion could be heard or not, or if there was an actual difference in sound quality between the hi-res and lowered-res sound? I.e. maybe there was a detectable difference in SQ but the question asked of the subjects explicitly referred to that noise only?

I'm confused.

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #6
Can smb remind me please if there was any AES article that would prove audibility of lowpass higher than 16 khz in music?  Is there a consensus in the AES circles on this matter?

(The question is not directly related to the paper in the OP. Sorry)

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #7
The test results show that the CD-quality A/D/A loop was undetectable at normal-to-loud listening levels, by any of the subjects, on any of the playback systems. The noise of the CD-quality loop was audible only at very elevated levels.

Sorry, does this mean they were testing whether this noise from the ADA conversion could be heard or not, or if there was an actual difference in sound quality between the hi-res and lowered-res sound? I.e. maybe there was a detectable difference in SQ but the question asked of the subjects explicitly referred to that noise only?

I'm confused.


It seems to me like they played hi-rez content downsampled to 16/44. That would be the fairest comparison, the high-rez master and the 16/44 version made directly from it.

And no one could tell the difference, what a huge surprise.

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #8
The title as well as the Abstract of this article seem only to refer to noise levels.  No one should be making any inferences to sound quality.  The conclusion of the article is that no one can distinguish a -96dB from a -120dB noise floor except at very high levers,  which everyone in this forum knows already.

Manuel

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #9
@ hushypushy:
You've rush-read my post and haven't answered my query.

I suspect only the OP can, or anyone else who's read the paper, unless I'm being completely dim.

To clarify, I'm unsure how to interpret the abstract as quoted, because it only talks about detecting the presence of the ADA loop, not general sound quality.

@ madman:
Okay.

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #10
Did they noise shape the 16-bit audio? Noise shaping, as we know, decreases the perceptual noise floor. This should improve the perceived quality of the 16/44 content.

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #11
The several comments about lack of inferences on sound quality do not fit either the title or the quote. It says no one can tell whether straight SACD or resample audio is being played. When no one can correctly guess whether it is  SACD or resample to CD spec audio, the only rational conclusion is that there is no audible difference in sound quality.

It says there is an audible difference is noise, but only when the volume is turned up very high. The only thing really unclear is “what noise?” Is it the CD noise floor, which cannot be heard in well mastered recordings except during very low level passages (played very loudly) , or is there some unusual (not to be expected in a normal CD) noise from their 16-bit/44.1-kHz "bottleneck."?

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #12
Sorry, I haven't had a whole lot of time to summarize the paper. I just wanted to be the first one to get the scoop

The paper is rather light on details of the setup. They don't list the speakers they use, or the DAC/ADC they use, or the analog gain stage they use to match levels. That's pretty disconcerting, but I'm unsure exactly how much that matters, given how many people were involved in testing. (They also tested at several different venues.) I'd suspect that nobody would be placated if they were more detailed in their equipment descriptions. They used an ABX CS-5 for switching. They tested the quantization level of 16/44 by ABXing the system with no music playing at +14db above the normal listening volume. When playing real music at that level, they described it as "unpleasantly (often unbearably) loud".

The paper concludes with the note that the high res releases sounded much, much better than the same music on CD, for well-known mastering reasons. So it ends on a surprisingly pro-high-res note. I'm actually more psyched up to buy SACD and DVD-A now compared to before I read it. I certainly haven't given much attention to high res releases, partly due to the copy restrictions, but partly because I always figured it was all bollocks and I might as well stick with CD.

In summary, I don't think this test will convince any audiophiles about the uselessness of high res. Some people will always come up with a bullshit excuse to believe what they want to believe, and there are a lot of people who have blind faith in SACD/DVD-A. But it could give producers more leeway to release well-mastered stuff on CD instead of more obscure formats. Or it could convince more people to get high res players to listen to better mastered stuff. Either way, listeners win.

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #13
They tested the quantization level of 16/44 by ABXing the system with no music playing
Was this in addition to ABX tests on the music, or were they really just testing what people thought about the noise? Why would they bother with the noise? It is easy to demonstrate the difference between quantization noise under those conditions (no music, high volume level) and no quantization noise because of good noise shaped dither, but who is going to avoid dithering during mastering anyway? i.e. what is the point of the test? Is there any way it relates to differences between SACD and CD, or were they just exploring a difference from CDs and their setup i.e. they didn't (noise shape) dither the 16/44.1 reduced version?


the high res releases sounded much, much better than the same music on CD, for well-known mastering reasons
This is still not clear as far as these tests go. The quote you provided above seemed to say that when this SACD mastered music was "passed through a 16-bit/44.1-kHz "bottleneck."" it could not be distinguished from the original.

If so, then the tests say there is nothing at all about SACD per se that is better -- except the mastering standards, which are quite irrelevant to the tests. Those differences from CD are arbitrary processing decisions and have nothing per se to do with the different technologies. If the mastering differences were not eliminated from the test, the tests would be as useless as those dual layer disks that are purposefully doctored to convince the `gullible that SACD is inherently better.

I think your statements are pretty clear, but I don't think everyone will take them the same way.

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #14
OK, I finally got the paper (written by the Boston Audio Society). Test setup was a hi-res media player (DVD-A and SACD), connected directly to the ABX switcher. The same player's output was connected to a 16/44.1 HHB (professional) cd-recorder (not listed, but clearly visible on a picture). Analog level matching was done only in the "lo-res" path, which was also fed to the ABX switcher. They did listen to music material:
Quote
Many types of music and voice signals were included in the sources, from classical (choral, chamber, piano, orchestral) to jazz, pop, and rock music. The subjects included men and women of widely varying ages, acuities, and levels of musical and audio experience; many were audio professionals or serious students of the art.
Quote
The test results for the detectability of the 16/44.1 loop on SACD/DVD-A playback were the same as chance: 49.82%. There were 554 trials and 276 correct answers. The sole exceptions were for the condition of no signal and high system gain, when the difference in noise floors of the two technologies, old and new, was readily audible.
The high system gain was about 14 dB. At that level, the noise difference became noticable, but music at the same play-back level was found to be "unpleasantly (often unbearably) loud".

ps:
Quote
...it is a rare playback venue that is quiet enough to reveal the 16-bit noise floor of our A/D/A loop—which has no noise shaping and was therefore less than optimal in this regard.

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #15
Sounds like more than just quantization noise was introduced with their CDDA "bottleneck".  Because of this and beyond the fact that they jacked the volume to an unrealistic level to reveal the differences I'd say the evidence is even more damning that hi-res formats provide no tangible benefit when dealing with 2-channel audio.

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #16
It implies the best and most convenient content available to consumers would be the SACD / DVD-A versions, recorded onto CD-R!

(because of the "well-known mastering reasons" - i.e. commercial CD releases are trashed with too much DRC, SACD/DVD-A sometimes are not)

I like the implication (which I agree with entirely) that you can make a better CD at home (using a good source, and a CD recorder's analogue input) than you can buy in a shop. It's quite sad.

Cheers,
David.

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #17
I think, the result of this paper and tests are clear and not surprising due to previous experiences, knowledge and tests.

Maybe you wonder about their "final conclusion", that DVD-A/SACD offer quality benefits compared to corresponding CDs, due to mastering reasons.
Well, simple answer: Though the AES is more or less independent and working scientific,
it is with every science, science needs money from the industry (and vice versa the industry needs sometimes researches by scientists), so the performers of that study have printed this sentence to their paper, to have something compromising sounding towards their industry... (without changing facts, truth etc. !), though Joe Average and the mass-magazines might focus only on this sentence, and change it only a litlte bit to "DVD-A/SACD proven to be superior to CDs !"


Don't laugh, a writer for HiFi-magazines in Germany has done already this in past, he conducted (a very poor conducted) similar test, and though his results, which he published as some kind of diploma work, indicated that CD and high-rez are on par,
but he managed to cite his diploma thesis falsely in magazines he wrote later for, writing to the audiophile masses, "a" diploma work had shown, hi-rez is superior (without mentioning, it was his own crappy diploma work/thesis) lolololol.

In audiophile internet forums, this guy, his diploma thesis, and his magazine articles were torn "in the air", even by audiophile guys !!

At least, this episode has shown to me, that even "audiophiles" (nothing wrong with being audiophile, we all are!) know about the marketing crap written in certain HiFi-magazines, if they are informed eg. by internet. Not everybody buys, what's advertized in those magazines.

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #18
The paper concludes with the note that the high res releases sounded much, much better than the same music on CD, for well-known mastering reasons. So it ends on a surprisingly pro-high-res note. I'm actually more psyched up to buy SACD and DVD-A now compared to before I read it. I certainly haven't given much attention to high res releases, partly due to the copy restrictions, but partly because I always figured it was all bollocks and I might as well stick with CD.


Theres really no need to move to an entirely new format though. Just more push for studios to release properly mastered CDs on labels like mofi or whatever.

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #19
The paper concludes with the note that the high res releases sounded much, much better than the same music on CD, for well-known mastering reasons. So it ends on a surprisingly pro-high-res note. I'm actually more psyched up to buy SACD and DVD-A now compared to before I read it. I certainly haven't given much attention to high res releases, partly due to the copy restrictions, but partly because I always figured it was all bollocks and I might as well stick with CD.



Sorry, but hi rez is no guarantee of good mastering.  I know for a fact that there are DVD-As out there that have the highly restricted dynamic range of modern CDs (Yes 'Fragile', anyone?).  You're still taking your chances.

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #20
Sorry, but hi rez is no guarantee of good mastering.  I know for a fact that there are DVD-As out there that have the highly restricted dynamic range of modern CDs (Yes 'Fragile', anyone?).  You're still taking your chances.

Agreed, but right now, the chances are not so bad. It all depends on how many producers do wind up remastering. Allegedly most of them have been doing it, but that could change in the future.

Moreover, if everybody really believed that the higher res is the ultimate reason of improved quality, then the perceived benefit of improved mastering (which is the real reason for improved quality) could be drastically reduced.

The myth of high res quality directly hurts consumers by increasing the risk of a shoddy mastering product.

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #21
So I've been monitoring Audio Asylum's thread on this (yeah, I know, bad idea). They do offer a few good criticisms.
  • JA asserted that the high res player they used, a Pioneer 563A, does not have any better of a dynamic range with high res recordings compared to CDs. Pioneer sez it's 108db, but hey, maybe it is lower in reality. It's asserted that this could obscure details in the high res listening, while still being low enough that the added quantization stage at 16/44 increases the noise level enough for audibility when listening to silence at loud levels.
  • Like I said, virtually no details are provided on the exact equipment used. The BAS website mentions the 563A in passing in an old article describing the test.
  • The statistical analysis is rather shoddy. The test results are not broken down by listener, or listening location, or by almost anything else. Type A and B error is not defined. The null hypothesis is not defined.
  • Musical selections are not listed.
  • Listeners are not described in terms of experience in any detail.

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #22
So I've been monitoring Audio Asylum's thread on this (yeah, I know, bad idea). They do offer a few good criticisms.
  • JA asserted that the high res player they used, a Pioneer 563A, does not have any better of a dynamic range with high res recordings compared to CDs. Pioneer sez it's 108db, but hey, maybe it is lower in reality. It's asserted that this could obscure details in the high res listening, while still being low enough that the added quantization stage at 16/44 increases the noise level enough for audibility when listening to silence at loud levels.
  • Like I said, virtually no details are provided on the exact equipment used. The BAS website mentions the 563A in passing in an old article describing the test.
  • The statistical analysis is rather shoddy. The test results are not broken down by listener, or listening location, or by almost anything else. Type A and B error is not defined. The null hypothesis is not defined.
  • Musical selections are not listed.
  • Listeners are not described in terms of experience in any detail.


Too bad.  I guess we'll have to wait for Atkinson & Co to perform the slam-dunk scientific demonstration that hi-rez really audibly matters 

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #23
>>    * JA asserted that the high res player they used, a Pioneer 563A, does not have any better of a dynamic range with high res recordings compared to CDs. Pioneer sez it's 108db, but hey, maybe it is lower in reality. It's asserted that this could obscure details in the high res listening, while still being low enough that the added quantization stage at 16/44 increases the noise level enough for audibility when listening to silence at loud levels.

These are not good criticisms. Just pro-forma nitpicking, so it appears something is wrong with the experiment. (There has to be, right?) There is nothing wrong with it. We used more than one hi-rez player. The noise floor of the venue was incredibly low. The loud levels were very loud. The source material --- lots of it, the widest range we could find, samplers, special demo cuts --- was extremely quiet in its noise floors.

>>    * Like I said, virtually no details are provided on the exact equipment used. The BAS website mentions the 563A in passing in an old article describing the test.

It does not matter what we used, since we degraded a hi-rez system and no one heard any difference, ever, regardless. We could have used an even better-quality 16/44 loop. But perhaps a genuinely compelling reason will arise and we will list the gear. The experiment was expanded to several other venues, including serious tweak systems, recording studios, and the like. No difference in the ability of the listeners to hear the "degradation."


>>    * The statistical analysis is rather shoddy. The test results are not broken down by listener, or listening location, or by almost anything else. Type A and B error is not defined. The null hypothesis is not defined.

What could this mean? Of course we spoke to statisticians, who were unanimous in saying it was a straightforward test of detectability, yes or no. Either some listeners heard a difference or not. No one did. No one came close. Any listener, any venue, any material. We did go over the results sorting by hearing bandwidth, sex, age, and experience. You will have to read the paper to be convinced, or not. If you really cannot afford to buy it, email me at drmoran@aol.com.


>>    * Musical selections are not listed.

Made no difference. This kind of thing is just to try to poke holes in an ironclad coin-flip result. We got a very wide range of material, types, instruments, vocals, genres, etc. Special stuff made for hi-rez demos, etc. etc.


>>    * Listeners are not described in terms of experience in any detail.

Not so --- just not enough detail for some who simply cannot believe the results.


David Moran

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe

Reply #24
>>    * JA asserted that the high res player they used, a Pioneer 563A, does not have any better of a dynamic range with high res recordings compared to CDs. Pioneer sez it's 108db, but hey, maybe it is lower in reality. It's asserted that this could obscure details in the high res listening, while still being low enough that the added quantization stage at 16/44 increases the noise level enough for audibility when listening to silence at loud levels.

These are not good criticisms. Just pro-forma nitpicking, so it appears something is wrong with the experiment. (There has to be, right?) There is nothing wrong with it. We used more than one hi-rez player. The noise floor of the venue was incredibly low. The loud levels were very loud. The source material --- lots of it, the widest range we could find, samplers, special demo cuts --- was extremely quiet in its noise floors.

>>    * Like I said, virtually no details are provided on the exact equipment used. The BAS website mentions the 563A in passing in an old article describing the test.

It does not matter what we used, since we degraded a hi-rez system and no one heard any difference, ever, regardless. We could have used an even better-quality 16/44 loop. But perhaps a genuinely compelling reason will arise and we will list the gear. The experiment was expanded to several other venues, including serious tweak systems, recording studios, and the like. No difference in the ability of the listeners to hear the "degradation."



The genuinely compelling reason for a detailed methods section in any scientific paper, is so the work can at least theoretically be reproduced.  And as you've seen, the subjectivist camp/Audio Asylum inmates will be all over you if you don't document everything scrupulously.



Quote
>>    * The statistical analysis is rather shoddy. The test results are not broken down by listener, or listening location, or by almost anything else. Type A and B error is not defined. The null hypothesis is not defined.

What could this mean?


It suggest you did not publish the Type I and Type II error statistics associated with your tests.  Most scientific papers that involve statistics at least include a Type I error (p value).


Quote
Of course we spoke to statisticians, who were unanimous in saying it was a straightforward test of detectability, yes or no. Either some listeners heard a difference or not. No one did. No one came close. Any listener, any venue, any material. We did go over the results sorting by hearing bandwidth, sex, age, and experience. You will have to read the paper to be convinced, or not. If you really cannot afford to buy it, email me at drmoran@aol.com.


I'd be happy to buy it, but it appears to be not yet available for purchase.  I'd love to get a copy.

Quote
>>    * Musical selections are not listed.

Made no difference. This kind of thing is just to try to poke holes in an ironclad coin-flip result. We got a very wide range of material, types, instruments, vocals, genres, etc. Special stuff made for hi-rez demos, etc. etc.



>>    * Listeners are not described in terms of experience in any detail.

Not so --- just not enough detail for some who simply cannot believe the results.


Believe it or now, most of us here are very much on your 'side'.  Hydrogenaudio even has advocacy of blind tests built into it Terms of Use. 

It's just regrettable if a paper on this perennial hot topic failed to provide extensive method detail, because every omission will be used against it. And thus a topic that should have been laid to rest years ago, will remain 'hot'.

Perhaps you could set up a 'supplementary materials and methods' webpage to provide those details?
This is common in journals where printed space is at a premium (e.g., Science, Nature).