IPB

Welcome Guest ( Log In | Register )

What's the problem with double-blind testing?
mirrorsawlljk
post Oct 19 2005, 03:03
Post #1





Group: Members
Posts: 11
Joined: 15-October 05
Member No.: 25138



I happened to pick up an issue of stereophile at a record store I visited and I was pretty shocked to see a seemingly intelligent person in the correspondence section bashing double blind testing as being unreliable. I'm afraid I don't understand his angle of attack. I don't see how anything could be a more reliable test of sound quality differences than a properly conducted double blind listening test.

I'm almost afraid to read the rest of the magazine if this is the kind of letter they think is worth publishing. Is there an audio magazine that isn't filled with this kind of thinking?
Go to the top of the page
+Quote Post
 
Start new topic
Replies
duff
post Oct 21 2005, 16:48
Post #2





Group: Members
Posts: 38
Joined: 18-July 02
From: Los Angeles, CA
Member No.: 2666



QUOTE
Seems to be a lack of understanding of something fundamental here: if you encode something to a lossy format it *does* lose information, it's not "compressed" at all. You've actually thrown away a lot of information! How do you know that the auditory system doesn't find it *easier* to decode since there's less "data"?


Lossless and lossy compression schemes are just that, compression. It is in this sense that I mean that an mp3 file, for example, is compressed..

The question of whether perceptual systems have an easier or harder time resolving stimuli that are impoverished (relative to a complementary uncompressed stimulus) is an empirical one that has been addressed in a variety of ways in perception research (mostly vision). I've been talking about ways to address this specifically with audio lossy schemes, and I'm presenting an idea regarding how this effect might not be addressed fully with an ABX paradigm that relies on decision tasks. A reaction time paradigm might be better suited to study the processing differences listeners likely experience.

The reason compressed data requires more effort on the part of whatever computational system is perceiving it, is because more inferential processing is required. If you are missing data in a signal, some mechanism in your brain must essentially interpolate, from the signal, the missing information.
Go to the top of the page
+Quote Post
stephanV
post Oct 21 2005, 17:29
Post #3





Group: Members
Posts: 394
Joined: 6-May 04
Member No.: 13932



QUOTE (duff @ Oct 21 2005, 05:48 PM)
The reason compressed data requires more effort on the part of whatever computational system is perceiving it, is because more inferential processing is required. If you are missing data in a signal, some mechanism in your brain must essentially interpolate, from the signal, the missing information.
*

Why? How can our brain even tell there is something missing? And why MUST it interpolate this data?

These are again merely assumptions on your part.

This post has been edited by stephanV: Oct 21 2005, 17:33


--------------------
"We cannot win against obsession. They care, we don't. They win."
Go to the top of the page
+Quote Post
duff
post Oct 21 2005, 20:06
Post #4





Group: Members
Posts: 38
Joined: 18-July 02
From: Los Angeles, CA
Member No.: 2666



QUOTE (stephanV @ Oct 21 2005, 04:29 PM)
QUOTE (duff @ Oct 21 2005, 05:48 PM)
The reason compressed data requires more effort on the part of whatever computational system is perceiving it, is because more inferential processing is required. If you are missing data in a signal, some mechanism in your brain must essentially interpolate, from the signal, the missing information.
*

Why? How can our brain even tell there is something missing? And why MUST it interpolate this data?

These are again merely assumptions on your part.
*



It's not that our brains "know" there is something missing, it's that relative to a less noisy signal, the brain has to do more work in order to generate the best representation of the sound it can.

I will post studies that support a number of things I have claimed, including quite notably the neural dissociation of sensory and decision processes. The "assumptions" I'm making are rooted in my training as a cognitive psychologist who studies speech processing. Many of the things I've said are well supported in experimental studies...I'm not just making it up. Also, what I'm saying is testable. I'm just trying to explain the logic of why I believe a different methodology might be necessary to rescue at least some of the claims made by audiophiles. I'm not an audiophile.

QUOTE
Why does it not necessarily apply? Because the purpose of lossy audio compression is to throw away unperceivable parts of the audio. So if the sensors aren't missing anything, then why should it make any difference?


Again, the main point here is that fatigue effects caused by processing issues (not accessible to systems that verbally report differences in ABX listening tests) could have residual effects on listeners in more long term ways. Reaction time experiments could verify this processing claim I'm making, and a positive result (where an ABX test shows nothing) could indicate a relationship between perceptual processing and vague ideas of discomfort experienced by some listeners of compressed audio.

This post has been edited by duff: Oct 21 2005, 20:19
Go to the top of the page
+Quote Post
stephanV
post Oct 21 2005, 20:21
Post #5





Group: Members
Posts: 394
Joined: 6-May 04
Member No.: 13932



QUOTE (duff @ Oct 21 2005, 09:06 PM)
QUOTE

Why? How can our brain even tell there is something missing? And why MUST it interpolate this data?

These are again merely assumptions on your part.
*


It's not that our brains "know" there is something missing, it's that relative to a less noisy signal, the brain has to do more work in order to generate the best representation of the sound it can.

Now you are saying something completely different. I don't see how our brain would try to represent anything else than what it is perceiving. I don't see how a wave form produced by an mp3 decoder is conceptually different from one might encounter in other situations. But ok, I will wait for studies.

This post has been edited by stephanV: Oct 21 2005, 20:24


--------------------
"We cannot win against obsession. They care, we don't. They win."
Go to the top of the page
+Quote Post
bryant
post Oct 21 2005, 22:53
Post #6


WavPack Developer


Group: Developer (Donating)
Posts: 1292
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



QUOTE (stephanV @ Oct 21 2005, 11:21 AM)
QUOTE (duff @ Oct 21 2005, 09:06 PM)
QUOTE

Why? How can our brain even tell there is something missing? And why MUST it interpolate this data?

These are again merely assumptions on your part.
*


It's not that our brains "know" there is something missing, it's that relative to a less noisy signal, the brain has to do more work in order to generate the best representation of the sound it can.

Now you are saying something completely different. I don't see how our brain would try to represent anything else than what it is perceiving. I don't see how a wave form produced by an mp3 decoder is conceptually different from one might encounter in other situations. But ok, I will wait for studies.
*


You have to keep in mind how MP3 encoders (and other lossy encoders) work. To say they "throw away" information is true, but it doesn't make it clear what they are really doing. What they do is reduce the resolution at certain frequencies and times (which is how they save space) and this has the effect of adding quantization noise to the signal. The idea is to essentially add noise to the signal as close as you can get to (but still below) the threshold where it's directly, consciously, audible. What's surprising is that this can be a relatively high level of noise (which is why these lossy codecs work so well). You can easily do a test where you subtract an original music signal from its encoded version. You might be surprised how much noise is added.

So, if we're adding noise to a signal that someone is trying to interpret, it's pretty obvious that it's going to require more processing. Do you find it easier to understand speech in a crowded bar or a quiet room? This is why they put up acoustic panels in auditoriums; too much reverberation acts like noise and makes speech more difficult to understand. This isn't really debatable.

What is in question is whether or not noise that is below the threshold where you can actually hear it still has an effect on processing, and what research is showing over and over is that whether or not something is directly perceived has very little to do with the underlying processing. The brain is processing the sounds into a "reality" for your conscious mind to act on, and does not want to bother you with unimportant details that might distract you. What if every stimuli registered by your senses was consciously perceived!!

Imagine walking blindfolded into a large room with sound reflective walls. You instantly become aware of the rough dimensions of the room, but you don't hear every echo of every footstep. A blind person would be able to "hear" in much greater detail, and in fact claim to be able to almost "see" the room. This is the kind of thing that the subconscious mind is constantly doing for us in the background, and what finally gets presented to our conscious mind is rather independent of what processing was required to achieve it. I admit that without at least a working knowledge of the literature, this may seem counterintuitive.

Another argument that was made is that if this extra processing automatic and we are not aware of it, then maybe it has no other effect. The problem with this is that the brain (and all parts of the body) don't work that way. The processing consumes resources (they actually measure this activity by measuring the radiation it generates) and all the body's systems try to minimize resource use. If I ask to you perform a math calculation in your head you would have to concentrate to do it (and you might refuse). If I asked you to do calculations over and over you would probably get tired and annoyed. Just because other processing is unconscious does not mean it's any less taxing, and the brain will always attempt to minimize it.

BTW, I'll clear up one minor error Greg made in his information comparison of a 128 kbps MP3 and the original wav. To compare the amount of real information stored you have to compare the size of the MP3 with a losslessly compressed wav, so it's really more like throwing away 80% of the information instead of 90%. This is more correct from an information theory standpoint.

The reason that I linked to the first of those two papers is that I find this field fascinating from a scientific viewpoint and I suspect that some HA members might be less aware of the richness of research discoveries in this area (compared to some other "harder" scientific realms).

The reason I linked to the second paper is that it directly relates to the auditory system, and points out a case where subconscious perception of a certain auditory stimuli turns out to be much more sensitive than what is consciously perceived. What could be more closely related to this discussion? I can imagine a version of the experiment where levels of noise might be added that, even when well below the threshold of conscious audibility, would have an effect on the measured subconscious perception of the timing. That would be the end of the discussion! That would prove that distortions below the threshold of conscious perception can effect how we hear music (at least to the somewhat open-minded).

Finally, and this was another motivation for me, there seems to be a tendancy here to oversimplify things that really are not that well understood. I am not trying to push any agenda and I am not an audiophile, but I am familiar enough with the scientific research to know that some of the claims often made here and taken as fact (at times in very condescending language) are simply not well supported at this time, and may in some cases turn out to be flatly wrong. I think it's fine if people have different opinions, and the free and lively interchange of ideas is what this board is all about. But I don't think it would hurt for a slightly more flexible attitude to prevail (and it would certainly reduce the possibility of many egg covered faces in the future).

Have a good weekend! smile.gif

edit: grammer

This post has been edited by bryant: Oct 21 2005, 22:57
Go to the top of the page
+Quote Post

Posts in this topic
- mirrorsawlljk   What's the problem with double-blind testing?   Oct 19 2005, 03:03
- - bubka   some people can actually detect specific codecs by...   Oct 19 2005, 03:07
- - TheQat   QUOTE (mirrorsawlljk @ Oct 18 2005, 06:03 PM)...   Oct 19 2005, 03:27
|- - singaiya   QUOTE (TheQat @ Oct 18 2005, 06:27 PM)QUOTE (...   Oct 19 2005, 04:31
|- - Yaztromo   QUOTE (TheQat @ Oct 19 2005, 03:27 AM)Edit: H...   Oct 19 2005, 23:05
- - Tahnru   The closest thing I have seen to a legitimate crit...   Oct 19 2005, 04:29
- - Axon   The Audio Critic is notably pro-DBT.   Oct 19 2005, 07:06
|- - PoisonDan   QUOTE (Axon @ Oct 19 2005, 08:06 AM)The Audio...   Oct 19 2005, 12:18
- - onthejazz   Interesting publication. I like it, too bad its no...   Oct 19 2005, 10:03
|- - Donunus   The writer of the letter in stereophile is incorre...   Oct 19 2005, 11:00
|- - Danimal   QUOTE (Donunus @ Oct 19 2005, 05:00 AM)The wr...   Oct 19 2005, 20:20
|- - Donunus   QUOTE (Danimal @ Oct 20 2005, 03:20 AM)QUOTE ...   Oct 20 2005, 01:51
|- - stephanV   QUOTE (Donunus @ Oct 20 2005, 02:51 AM)Well, ...   Oct 20 2005, 08:48
||- - bryant   QUOTE (stephanV @ Oct 19 2005, 11:48 PM)QUOTE...   Oct 20 2005, 18:11
|||- - Pio2001   QUOTE (bryant @ Oct 20 2005, 07:11 PM)To tell...   Oct 20 2005, 20:48
||||- - singaiya   Thanks for the papers, David. The first one is a g...   Oct 20 2005, 22:13
||||- - duff   QUOTE (Pio2001 @ Oct 20 2005, 07:48 PM)QUOTE ...   Oct 20 2005, 22:52
||||- - stephanV   QUOTE (duff @ Oct 20 2005, 11:52 PM)It's ...   Oct 20 2005, 23:11
|||||- - duff   QUOTE Might be... or might be not. You can't c...   Oct 20 2005, 23:36
|||||- - KikeG   QUOTE (duff @ Oct 20 2005, 11:36 PM)I think i...   Oct 21 2005, 00:00
||||||- - rjamorim   QUOTE (KikeG @ Oct 20 2005, 09:00 PM)Also, ps...   Oct 21 2005, 00:15
||||||- - duff   QUOTE (KikeG @ Oct 20 2005, 11:00 PM)QUOTE (d...   Oct 21 2005, 00:29
||||||- - rjamorim   QUOTE (duff @ Oct 20 2005, 09:29 PM)I'm n...   Oct 21 2005, 01:03
||||||- - Mike Giacomelli   QUOTE (duff @ Oct 20 2005, 04:29 PM)The audit...   Oct 21 2005, 05:56
||||||- - KikeG   QUOTE (duff @ Oct 21 2005, 12:29 AM)From the ...   Oct 21 2005, 08:10
|||||- - stephanV   QUOTE (duff @ Oct 21 2005, 12:36 AM)I think i...   Oct 21 2005, 08:34
|||||- - Woodinville   QUOTE (stephanV @ Oct 20 2005, 11:34 PM)Sorry...   Oct 21 2005, 21:19
||||- - krabapple   QUOTE (duff @ Oct 20 2005, 04:52 PM)QUOTE (Pi...   Oct 21 2005, 05:25
||||- - antz   QUOTE (duff @ Oct 20 2005, 10:52 PM)QUOTE (Pi...   Oct 21 2005, 13:39
||||- - Woodinville   QUOTE (antz @ Oct 21 2005, 04:39 AM)Seems to ...   Oct 21 2005, 21:20
|||- - stephanV   QUOTE (bryant @ Oct 20 2005, 07:11 PM)It turn...   Oct 20 2005, 21:43
||- - Donunus   QUOTE (stephanV @ Oct 20 2005, 03:48 PM)QUOTE...   Oct 21 2005, 04:10
||- - stephanV   QUOTE (Donunus @ Oct 21 2005, 05:10 AM)I do n...   Oct 21 2005, 08:43
||- - user   I recall, we have had successful abx tests even he...   Oct 21 2005, 09:20
||- - Pio2001   QUOTE (user @ Oct 21 2005, 10:20 AM)I recall,...   Oct 21 2005, 12:18
|||- - ff123   QUOTE (Pio2001 @ Oct 21 2005, 03:18 AM)QUOTE ...   Oct 21 2005, 17:43
||- - krabapple   QUOTE (user @ Oct 21 2005, 03:20 AM)I recall,...   Oct 21 2005, 21:54
|- - Cartoon   QUOTE (Donunus @ Oct 20 2005, 02:51 AM)I do h...   Nov 12 2005, 13:40
- - KikeG   The "goosebump", emotional factor can be...   Oct 19 2005, 11:54
|- - Donunus   QUOTE (KikeG @ Oct 19 2005, 06:54 PM)The ...   Oct 19 2005, 12:06
|- - kjoonlee   QUOTE (Donunus @ Oct 19 2005, 08:06 PM)actual...   Oct 19 2005, 18:31
||- - Woodinville   QUOTE (kjoonlee @ Oct 19 2005, 09:31 AM)QUOTE...   Oct 21 2005, 01:27
||- - duff   QUOTE If you consider that we are wired to detect ...   Oct 21 2005, 01:38
||- - Woodinville   QUOTE (duff @ Oct 20 2005, 04:38 PM)So we are...   Oct 21 2005, 21:15
||- - duff   QUOTE (Woodinville @ Oct 21 2005, 08:15 PM)QU...   Oct 21 2005, 23:43
||- - Woodinville   QUOTE (duff @ Oct 21 2005, 02:43 PM)When I sa...   Oct 22 2005, 08:00
|- - Lyx   QUOTE (Donunus @ Oct 19 2005, 01:06 PM)QUOTE ...   Oct 20 2005, 06:07
|- - Donunus   QUOTE (Lyx @ Oct 20 2005, 01:07 PM)QUOTE (Don...   Oct 20 2005, 16:51
|- - Lyx   QUOTE (Donunus @ Oct 20 2005, 05:51 PM)QUOTE ...   Oct 20 2005, 21:10
- - KikeG   Well, I think expectation effects in listening tes...   Oct 19 2005, 19:33
|- - krabapple   QUOTE (KikeG @ Oct 19 2005, 10:33 AM)Well, I ...   Oct 19 2005, 20:12
- - krabapple   QUOTE (mirrorsawlljk @ Oct 18 2005, 06:03 PM)...   Oct 19 2005, 20:06
- - Halcyon   There is nothing inherently bad about double blind...   Oct 19 2005, 20:29
- - ff123   Tests that try to distinguish very small effects *...   Oct 19 2005, 20:53
|- - krabapple   QUOTE (ff123 @ Oct 19 2005, 11:53 AM)Tests th...   Oct 19 2005, 23:40
|- - ff123   QUOTE (krabapple @ Oct 19 2005, 02:40 PM)Inte...   Oct 20 2005, 05:38
- - fcmts   There is another problem similar to wine blind tes...   Oct 19 2005, 22:15
- - stephanV   I don't see how that is a problem. The goal of...   Oct 19 2005, 22:22
- - duff   One distinction relevant to this issue is the diff...   Oct 20 2005, 23:00
- - duff   QUOTE The opposite of "blind", I suppose...   Oct 21 2005, 01:27
- - Jun-Dai   I think it's pretty clear that double-blind te...   Oct 21 2005, 07:22
- - KikeG   For helping doing long-term, casual listening-like...   Oct 21 2005, 08:30
- - duff   QUOTE Seems to be a lack of understanding of somet...   Oct 21 2005, 16:48
|- - stephanV   QUOTE (duff @ Oct 21 2005, 05:48 PM)The reaso...   Oct 21 2005, 17:29
||- - duff   QUOTE (stephanV @ Oct 21 2005, 04:29 PM)QUOTE...   Oct 21 2005, 20:06
||- - stephanV   QUOTE (duff @ Oct 21 2005, 09:06 PM)QUOTE Wh...   Oct 21 2005, 20:21
||- - bryant   QUOTE (stephanV @ Oct 21 2005, 11:21 AM)QUOTE...   Oct 21 2005, 22:53
||- - rjamorim   QUOTE (bryant @ Oct 21 2005, 07:53 PM)But I d...   Oct 21 2005, 23:12
||- - stephanV   QUOTE (bryant @ Oct 21 2005, 11:53 PM)So, if ...   Oct 22 2005, 00:00
||- - ChiGung   QUOTE (bryant @ Oct 21 2005, 10:53 PM)What...   Oct 22 2005, 00:25
|||- - bryant   QUOTE (ChiGung @ Oct 21 2005, 03:25 PM)I have...   Oct 22 2005, 23:39
|||- - ChiGung   just thinking -i do get all sorts of weird sensati...   Oct 23 2005, 00:34
||- - Woodinville   QUOTE (bryant @ Oct 21 2005, 01:53 PM)So, if ...   Oct 22 2005, 07:52
||- - bryant   QUOTE (Woodinville @ Oct 21 2005, 10:52 PM)QU...   Oct 22 2005, 23:32
||- - Woodinville   QUOTE (bryant @ Oct 22 2005, 02:32 PM)The onl...   Oct 24 2005, 21:51
|- - Lyx   QUOTE (duff @ Oct 21 2005, 05:48 PM)QUOTE See...   Oct 21 2005, 17:35
|- - krabapple   QUOTE (duff @ Oct 21 2005, 10:48 AM)QUOTE See...   Oct 21 2005, 22:01
- - duff   QUOTE It sounds counterintuitive that my brain wou...   Oct 22 2005, 00:30
|- - ChiGung   QUOTE (duff @ Oct 22 2005, 12:30 AM)Frankly, ...   Oct 22 2005, 00:47
||- - duff   QUOTE (ChiGung @ Oct 21 2005, 11:47 PM)QUOTE ...   Oct 22 2005, 01:14
||- - ChiGung   QUOTE (duff @ Oct 22 2005, 01:14 AM)The reduc...   Oct 22 2005, 01:26
|- - Woodinville   QUOTE (duff @ Oct 21 2005, 03:30 PM)Filtering...   Oct 22 2005, 08:07
- - ChiGung   A simplistic example - a sine wave of exactly 3122...   Oct 22 2005, 00:59
|- - Woodinville   QUOTE (ChiGung @ Oct 21 2005, 03:59 PM)A simp...   Oct 22 2005, 08:08
|- - ChiGung   QUOTE (Woodinville @ Oct 22 2005, 08:08 AM)QU...   Oct 22 2005, 14:09
- - ChiGung   -Sorry for flooding a bit there. Tasty subject   Oct 22 2005, 01:01
- - duff   So sorry for the long delay. I didn't forget a...   Nov 10 2005, 20:10
|- - Woodinville   QUOTE (duff @ Nov 10 2005, 11:10 AM)Woodinsid...   Nov 10 2005, 20:23
- - duff   I don't understand why you would want to maint...   Nov 11 2005, 00:41
|- - Woodinville   QUOTE (duff @ Nov 10 2005, 03:41 PM)Somehow, ...   Nov 11 2005, 21:52
|- - Pio2001   QUOTE (duff @ Nov 11 2005, 01:41 AM)The ABX p...   Nov 12 2005, 03:00
|- - Woodinville   QUOTE (Pio2001 @ Nov 11 2005, 06:00 PM)I can ...   Nov 12 2005, 09:33
|- - user   There aren't theoretical flaws with abx or DB ...   Nov 12 2005, 10:52
- - KikeG   Sorry, you still have provided no evidence that wh...   Nov 11 2005, 14:34
|- - Woodinville   QUOTE (KikeG @ Nov 11 2005, 05:34 AM)Sorry, y...   Nov 11 2005, 21:56
|- - Pio2001   QUOTE (Woodinville @ Nov 11 2005, 10:56 PM)Su...   Nov 12 2005, 02:32
- - duff   Let's back up a moment... There are at least ...   Nov 11 2005, 18:37
- - KikeG   About the first point, your degraded examples are ...   Nov 11 2005, 19:57
- - duff   QUOTE Well, if it's inaudible, there's a g...   Nov 11 2005, 21:19
- - Woodinville   QUOTE (duff @ Nov 11 2005, 12:19 PM)In other ...   Nov 11 2005, 21:59
4 Pages V   1 2 3 > » 


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 1st October 2014 - 07:19