IPB

Welcome Guest ( Log In | Register )

Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe, (hint: no surprises!) (bumped from 2007 in post #35)
Axon
post Sep 10 2007, 04:11
Post #1





Group: Members (Donating)
Posts: 1984
Joined: 4-January 04
From: Austin, TX
Member No.: 10933



"Audibility of a CD-Standard A/D/A Loop Inserted into High-Resolution Audio Playback". E. Brad Meyer and David R. Moran. JAES 55(9) September 2007. It's worth noting that members of the BAS wrote the paper and performed in the tests. You'll need an AES membership to access the article, so no link.

Abstract:
QUOTE
Claims both published and anecdotal are regularly made for audibly superior sound quality for two-channel audio encoded with longer word lengths and/or at higher sampling rates than the 16-bit/44.1-kHz CD standard. The authors report on a series of double-blind tests comparing the analog output of high-resolution players playing high-resolution recordings with the same signal passed through a 16-bit/44.1-kHz "bottleneck." The tests were conducted for over a year using different systems and a variety of subjects. The systems included expensive professional monitors and one high-end system with electrostatic loudspeakers and expensive components and cables. The subjects included professional recording engineers, students in a university recording program, and dedicated audiophiles. The test results show that the CD-quality A/D/A loop was undetectable at normal-to-loud listening levels, by any of the subjects, on any of the playback systems. The noise of the CD-quality loop was audible only at very elevated levels.


This post has been edited by Axon: Sep 10 2007, 04:13
Go to the top of the page
+Quote Post
 
Start new topic
Replies
Axon
post Sep 13 2007, 20:23
Post #2





Group: Members (Donating)
Posts: 1984
Joined: 4-January 04
From: Austin, TX
Member No.: 10933



So I've been monitoring Audio Asylum's thread on this (yeah, I know, bad idea). They do offer a few good criticisms.
  • JA asserted that the high res player they used, a Pioneer 563A, does not have any better of a dynamic range with high res recordings compared to CDs. Pioneer sez it's 108db, but hey, maybe it is lower in reality. It's asserted that this could obscure details in the high res listening, while still being low enough that the added quantization stage at 16/44 increases the noise level enough for audibility when listening to silence at loud levels.
  • Like I said, virtually no details are provided on the exact equipment used. The BAS website mentions the 563A in passing in an old article describing the test.
  • The statistical analysis is rather shoddy. The test results are not broken down by listener, or listening location, or by almost anything else. Type A and B error is not defined. The null hypothesis is not defined.
  • Musical selections are not listed.
  • Listeners are not described in terms of experience in any detail.
Go to the top of the page
+Quote Post
krabapple
post Sep 14 2007, 22:56
Post #3





Group: Members
Posts: 2274
Joined: 18-December 03
Member No.: 10538



QUOTE (Axon @ Sep 13 2007, 15:23) *
So I've been monitoring Audio Asylum's thread on this (yeah, I know, bad idea). They do offer a few good criticisms.
  • JA asserted that the high res player they used, a Pioneer 563A, does not have any better of a dynamic range with high res recordings compared to CDs. Pioneer sez it's 108db, but hey, maybe it is lower in reality. It's asserted that this could obscure details in the high res listening, while still being low enough that the added quantization stage at 16/44 increases the noise level enough for audibility when listening to silence at loud levels.
  • Like I said, virtually no details are provided on the exact equipment used. The BAS website mentions the 563A in passing in an old article describing the test.
  • The statistical analysis is rather shoddy. The test results are not broken down by listener, or listening location, or by almost anything else. Type A and B error is not defined. The null hypothesis is not defined.
  • Musical selections are not listed.
  • Listeners are not described in terms of experience in any detail.


Too bad. I guess we'll have to wait for Atkinson & Co to perform the slam-dunk scientific demonstration that hi-rez really audibly matters rolleyes.gif
Go to the top of the page
+Quote Post
david moran
post Sep 19 2007, 22:15
Post #4





Group: Members
Posts: 2
Joined: 19-September 07
Member No.: 47190



>> * JA asserted that the high res player they used, a Pioneer 563A, does not have any better of a dynamic range with high res recordings compared to CDs. Pioneer sez it's 108db, but hey, maybe it is lower in reality. It's asserted that this could obscure details in the high res listening, while still being low enough that the added quantization stage at 16/44 increases the noise level enough for audibility when listening to silence at loud levels.

These are not good criticisms. Just pro-forma nitpicking, so it appears something is wrong with the experiment. (There has to be, right?) There is nothing wrong with it. We used more than one hi-rez player. The noise floor of the venue was incredibly low. The loud levels were very loud. The source material --- lots of it, the widest range we could find, samplers, special demo cuts --- was extremely quiet in its noise floors.

>> * Like I said, virtually no details are provided on the exact equipment used. The BAS website mentions the 563A in passing in an old article describing the test.

It does not matter what we used, since we degraded a hi-rez system and no one heard any difference, ever, regardless. We could have used an even better-quality 16/44 loop. But perhaps a genuinely compelling reason will arise and we will list the gear. The experiment was expanded to several other venues, including serious tweak systems, recording studios, and the like. No difference in the ability of the listeners to hear the "degradation."


>> * The statistical analysis is rather shoddy. The test results are not broken down by listener, or listening location, or by almost anything else. Type A and B error is not defined. The null hypothesis is not defined.

What could this mean? Of course we spoke to statisticians, who were unanimous in saying it was a straightforward test of detectability, yes or no. Either some listeners heard a difference or not. No one did. No one came close. Any listener, any venue, any material. We did go over the results sorting by hearing bandwidth, sex, age, and experience. You will have to read the paper to be convinced, or not. If you really cannot afford to buy it, email me at drmoran@aol.com.


>> * Musical selections are not listed.

Made no difference. This kind of thing is just to try to poke holes in an ironclad coin-flip result. We got a very wide range of material, types, instruments, vocals, genres, etc. Special stuff made for hi-rez demos, etc. etc.


>> * Listeners are not described in terms of experience in any detail.

Not so --- just not enough detail for some who simply cannot believe the results.


David Moran
Go to the top of the page
+Quote Post
Axon
post Sep 20 2007, 00:42
Post #5





Group: Members (Donating)
Posts: 1984
Joined: 4-January 04
From: Austin, TX
Member No.: 10933



Thank you for replying (and hopefully you'll stick around!)

QUOTE (david moran @ Sep 19 2007, 16:15) *
These are not good criticisms. Just pro-forma nitpicking, so it appears something is wrong with the experiment. (There has to be, right?) There is nothing wrong with it. We used more than one hi-rez player. The noise floor of the venue was incredibly low. The loud levels were very loud. The source material --- lots of it, the widest range we could find, samplers, special demo cuts --- was extremely quiet in its noise floors.
Well, duh! Of course it's nitpicking. That point isn't evidence of anything actually being wrong with the experiment - but it seems to me like it could potentially obscure real issues. Or, at the very least, the information is useful if another blind test would be attempted, in order to meet or exceed the technical specifications of the original test. (I'm not saying that I'd be doing it, but again, nice to know.)

QUOTE
It does not matter what we used, since we degraded a hi-rez system and no one heard any difference, ever, regardless. We could have used an even better-quality 16/44 loop. But perhaps a genuinely compelling reason will arise and we will list the gear. The experiment was expanded to several other venues, including serious tweak systems, recording studios, and the like. No difference in the ability of the listeners to hear the "degradation."
Why would you wait for a reason to arise before listing the gear? That isn't a rhetorical question, and I'm not trying to be confrontational. I just don't understand. A lot of the individual ABX tests conducted here have pretty detailed gear descriptions associated with them, so I figure an AES paper would do the same thing...

QUOTE
What could this mean? Of course we spoke to statisticians, who were unanimous in saying it was a straightforward test of detectability, yes or no. Either some listeners heard a difference or not. No one did. No one came close. Any listener, any venue, any material. We did go over the results sorting by hearing bandwidth, sex, age, and experience. You will have to read the paper to be convinced, or not. If you really cannot afford to buy it, email me at drmoran@aol.com.
I already emailed you (and Mr. Meyer) 3 days ago, actually. And I posted here when I first saw the complete article in the online JAES.

You're allowed to drag me back to the woodshed and beat me for calling the statistics "shoddy", if you want. Heck, I think I may have the least statistics experience of anybody commenting in this thread. What I meant to say was that the paper violated my expectations of how much detail is supposed to exist in the results/analysis. Based on other blind tests I've read, I would have expected to see clearly defined null hypotheses, the full listener responses, numerical analyses thereof, etc. Maybe some lip service to estimating the proportion of discriminators (which, honestly, really ought to be 1 anyway - but still, it's a subject worthy of debate, and I'd figure it's worth commenting over).

The results as they stand in the paper seemed to breeze through a lot of this very quickly, which surprised me. That doesn't diminish the results any - they are shockingly conclusive. They're just, well, short. And I think many audiophiles are using that as an excuse to ignore them completely. (Granted, though, most of them would probably ignore the paper anyway.)

QUOTE
Not so --- just not enough detail for some who simply cannot believe the results.
Alas, you're confusing me for somebody who doesn't believe the results.

I really do. They match my personal evaluation of SACD/DVD-A, they match my understanding of audio engineering and psychoacoustics, and the obviously large amount of work that went into the setup and listening, combined with the large number of listeners, makes them extremely compelling. And as the rest of this thread indicates you'd be hard-pressed to find anybody else here who would not agree. Enough of us have done our own personal ABX tests to understand how much effort goes into them, and what kind of results one can draw from them.

But when such a compelling result is established, why be stingy with so much information? You state that the ball's now in the pro-highres camp to show any sort of proof of audibility, but it seems like if anybody actually attempted another test like this, they'd want to know all of these things, in order to try to strengthen the sensitivity of the test. You spend a considerable amount of effort in stating that so many of these factors were carefully eliminated with the best testing conditions, and yet you don't actually say how you do it. "Trust us" is not a valid argument!

Already a lot of audiophiles (not me!) are using the lack of information in this paper to justify some sort of AES cabal that does not actually engage in peer review. Like krabapple said - why try and settle the matter for good, when you've thrown them such a juicy bone to chew on at the same time? Unless the paper was meant only for AES readers, not for audiophiles.

As far as I'm personally concerned, all of those issues are really of omission than of actual error. I just want to know the information. Or, I just want to know why it's not there. It's mainly because I'm curious. I'm not expecting to find any flaws in it. And I had the expectation that asking for information like this is an extremely reasonable request.

This post has been edited by Axon: Sep 20 2007, 05:22
Go to the top of the page
+Quote Post

Posts in this topic
- Axon   Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe   Sep 10 2007, 04:11
- - PoisonDan   Interesting, thanks!   Sep 10 2007, 07:08
- - hushypushy   Very interesting...people are still going to claim...   Sep 10 2007, 07:14
- - Kees de Visser   Thanks for the info. Any details about the A/D/A l...   Sep 10 2007, 12:16
- - LANjackal   Thanks for this   Sep 10 2007, 13:18
- - Bad Monkey   The test results show that the CD-quality A/D/A lo...   Sep 10 2007, 13:48
|- - hushypushy   QUOTE (Bad Monkey @ Sep 10 2007, 05:48) T...   Sep 10 2007, 17:15
- - kdo   Can smb remind me please if there was any AES arti...   Sep 10 2007, 13:58
- - Madman1153   The title as well as the Abstract of this article ...   Sep 10 2007, 17:57
- - Bad Monkey   @ hushypushy: You've rush-read my post and hav...   Sep 10 2007, 17:59
- - Canar   Did they noise shape the 16-bit audio? Noise shapi...   Sep 10 2007, 18:47
- - AndyH-ha   The several comments about lack of inferences on s...   Sep 10 2007, 19:34
- - Axon   Sorry, I haven't had a whole lot of time to su...   Sep 10 2007, 20:37
|- - Kees de Visser   OK, I finally got the paper (written by the Boston...   Sep 11 2007, 06:33
|- - dmckean   QUOTE (Axon @ Sep 10 2007, 12:37) The pap...   Sep 12 2007, 00:59
|- - krabapple   QUOTE (Axon @ Sep 10 2007, 15:37) The pap...   Sep 13 2007, 17:27
|- - Axon   QUOTE (krabapple @ Sep 13 2007, 11:27) So...   Sep 13 2007, 20:02
- - AndyH-ha   They tested the quantization level of 16/44 by ABX...   Sep 11 2007, 00:03
- - greynol   Sounds like more than just quantization noise was ...   Sep 11 2007, 06:53
- - 2Bdecided   It implies the best and most convenient content av...   Sep 11 2007, 12:26
|- - user   I think, the result of this paper and tests are cl...   Sep 11 2007, 13:53
- - Axon   So I've been monitoring Audio Asylum's thr...   Sep 13 2007, 20:23
|- - krabapple   QUOTE (Axon @ Sep 13 2007, 15:23) So I...   Sep 14 2007, 22:56
|- - david moran   >> * JA asserted that the high res playe...   Sep 19 2007, 22:15
|- - krabapple   QUOTE (david moran @ Sep 19 2007, 17:15) ...   Sep 19 2007, 23:52
||- - david moran   Okay. Thank you all for your thoughtful responses,...   Sep 20 2007, 22:40
|||- - krabapple   QUOTE (david moran @ Sep 20 2007, 17:40) ...   Sep 22 2007, 06:48
|||- - Kees de Visser   QUOTE (david moran @ Sep 20 2007, 23:40) ...   Sep 22 2007, 08:52
|||- - spockep   QUOTE (david moran @ Sep 20 2007, 17:40) ...   Sep 22 2007, 13:37
||- - bernlin2000   QUOTE (krabapple @ Sep 19 2007, 16:52) Be...   Dec 20 2012, 10:50
|- - Axon   Thank you for replying (and hopefully you'll s...   Sep 20 2007, 00:42
- - dekkersj   I contacted the authors and they replied that they...   Sep 20 2007, 20:58
- - dekkersj   Thanks David, Good to talk to you this way I ...   Sep 20 2007, 22:49
- - Axon   What they said. Thanks for responding. Clearly i...   Sep 23 2007, 04:01
|- - 2Bdecided   QUOTE (Axon @ Sep 23 2007, 04:01) For eve...   Sep 24 2007, 11:39
- - usernaim   I'm not an advocate of hi-rez, but I wanted to...   May 28 2012, 18:43
|- - Arnold B. Krueger   QUOTE (usernaim @ May 28 2012, 13:43) I...   May 29 2012, 15:37
|- - 2Bdecided   QUOTE (Arnold B. Krueger @ May 29 2012, 15...   May 29 2012, 17:47
|- - Arnold B. Krueger   QUOTE (2Bdecided @ May 29 2012, 12:47) QU...   May 29 2012, 21:10
- - 2Bdecided   Arny, I'm on your side. The point is they clai...   May 31 2012, 10:17
|- - Arnold B. Krueger   QUOTE (2Bdecided @ May 31 2012, 05:17) Ar...   May 31 2012, 12:45
- - SoNic67   I am a fan of SACD recordings. Not necesarelly bec...   Jun 1 2012, 00:08
|- - 2Bdecided   QUOTE (SoNic67 @ Jun 1 2012, 00:08) Anywa...   Jun 1 2012, 11:40
|- - Arnold B. Krueger   QUOTE (2Bdecided @ Jun 1 2012, 06:40) QUO...   Jun 1 2012, 12:01
|- - SoNic67   QUOTE (2Bdecided @ Jun 1 2012, 06:40) You...   Jun 2 2012, 00:56
|- - Arnold B. Krueger   QUOTE (Sonic67)Everyone tells me that I can't ...   Jun 3 2012, 00:27
- - Wombat   QUOTE (2Bdecided @ May 29 2012, 18:47) Fa...   Jun 3 2012, 00:11
- - bernlin2000   QUOTE (Axon @ Sep 9 2007, 21:11) "Au...   Dec 20 2012, 10:44
- - DonP   One argument I would consider is that if any compo...   Dec 20 2012, 14:26
|- - sawdin   If the BAS testing was so flawed, why haven't ...   Jan 6 2013, 19:58
|- - Arnold B. Krueger   QUOTE (sawdin @ Jan 6 2013, 13:58) If the...   Jan 7 2013, 15:51
|- - drmoran@aol.com   QUOTE (Arnold B. Krueger @ Jan 7 2013, 09...   Jan 20 2013, 01:25
- - mzil   Dr. Moran, I think this test you used is brilliant...   Jan 20 2013, 06:01
|- - Arnold B. Krueger   QUOTE (mzil @ Jan 20 2013, 00:01) Dr. Mor...   Jan 22 2013, 19:02
|- - mzil   QUOTE (Arnold B. Krueger @ Jan 22 2013, 14...   Jan 22 2013, 22:50
- - eretsua   The difference in sound between vinyl and CDs, I t...   Jan 23 2013, 03:22
|- - Engelsstaub   QUOTE (eretsua @ Jan 22 2013, 20:22) The ...   Jan 23 2013, 04:28
||- - julf   QUOTE (Engelsstaub @ Jan 23 2013, 04:28) ...   Jan 23 2013, 08:33
||- - markanini   QUOTE (julf @ Jan 23 2013, 08:33) QUOTE (...   Jan 23 2013, 09:14
||- - julf   QUOTE (markanini @ Jan 23 2013, 09:14) Do...   Jan 23 2013, 10:54
||- - eretsua   Sigh, regardless of how well or how poorly anyone ...   Jan 23 2013, 11:27
||- - julf   QUOTE (eretsua @ Jan 23 2013, 11:27) Sigh...   Jan 23 2013, 12:10
|- - Arnold B. Krueger   QUOTE (eretsua @ Jan 22 2013, 21:22) The ...   Jan 24 2013, 19:45
- - markanini   That's like calling a prostitute a nun for hav...   Jan 24 2013, 06:40
- - 2Bdecided   No, I think he means use the vinyl as the source, ...   Jan 25 2013, 11:14
|- - Arnold B. Krueger   QUOTE (2Bdecided @ Jan 25 2013, 05:14) No...   Jan 25 2013, 13:17
|- - db1989   QUOTE (Arnold B. Krueger @ Jan 25 2013, 12...   Jan 25 2013, 13:50
||- - dhromed   QUOTE (db1989 @ Jan 25 2013, 13:50) Iím s...   Jan 25 2013, 14:02
|- - 2Bdecided   QUOTE (Arnold B. Krueger @ Jan 25 2013, 12...   Jan 25 2013, 14:01
- - db1989   Oh lawd! Why didnít I ever suspect before? Co...   Jan 25 2013, 14:12
- - greynol   eretsua is suggesting that a CD using the same mas...   Jan 25 2013, 17:47
- - eretsua   Greynol is correct. What I meant is that indeed th...   Jan 26 2013, 00:14
|- - krabapple   QUOTE (eretsua @ Jan 25 2013, 19:14) So t...   Jan 26 2013, 17:48
|- - Arnold B. Krueger   QUOTE (eretsua @ Jan 25 2013, 18:14) So t...   Jan 28 2013, 19:24
|- - greynol   QUOTE (Arnold B. Krueger @ Jan 28 2013, 10...   Jan 28 2013, 20:03
- - greynol   We have a saying around here: you can make your CD...   Jan 26 2013, 00:42
- - 2Bdecided   I'm totally lost. Anyway, while I'm happ...   Jan 28 2013, 14:04
|- - Nessuno   QUOTE (2Bdecided @ Jan 28 2013, 14:04) I...   Jan 28 2013, 15:25
|- - greynol   QUOTE (2Bdecided @ Jan 28 2013, 05:04) I...   Jan 28 2013, 19:54
|- - krabapple   QUOTE (2Bdecided @ Jan 28 2013, 09:04) I...   Jan 29 2013, 05:12
- - 2Bdecided   I was lost with who suggested we blind tested what...   Jan 29 2013, 10:45
- - jeffb   Re: vinyl and CDs... I enjoy both, and have over ...   Jun 14 2013, 23:57
- - greynol   Due to the two off-topic posts in the last 24 hour...   Jun 15 2013, 01:22


Closed TopicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 22nd September 2014 - 00:42