IPB

Welcome Guest ( Log In | Register )

What's the problem with double-blind testing?
mirrorsawlljk
post Oct 19 2005, 03:03
Post #1





Group: Members
Posts: 11
Joined: 15-October 05
Member No.: 25138



I happened to pick up an issue of stereophile at a record store I visited and I was pretty shocked to see a seemingly intelligent person in the correspondence section bashing double blind testing as being unreliable. I'm afraid I don't understand his angle of attack. I don't see how anything could be a more reliable test of sound quality differences than a properly conducted double blind listening test.

I'm almost afraid to read the rest of the magazine if this is the kind of letter they think is worth publishing. Is there an audio magazine that isn't filled with this kind of thinking?
Go to the top of the page
+Quote Post
 
Start new topic
Replies
KikeG
post Oct 19 2005, 11:54
Post #2


WinABX developer


Group: Developer
Posts: 1578
Joined: 1-October 01
Member No.: 137



The "goosebump", emotional factor can be caused by placebo effect, you can't be sure that is not the cause. People has used same principle (emotional response and the like) to say cables sound different, for example.

You can do long-term blind tests too, not just the quick-switch, short snippet, typical abx test.

This post has been edited by KikeG: Oct 19 2005, 11:55
Go to the top of the page
+Quote Post
Donunus
post Oct 19 2005, 12:06
Post #3





Group: Members
Posts: 226
Joined: 8-July 05
Member No.: 23210



QUOTE (KikeG @ Oct 19 2005, 06:54 PM)
The "goosebump", emotional factor can be caused by placebo effect, you can't be sure that is not the cause. People has used same principle (emotional response and the like) to say cables sound different, for example.

You can do long-term blind tests too, not just the quick-switch, short snippet, typical abx test.
*

What do you mean by long term abx? how do I do this? Play entire songs? actually can you give me a link that explains placebo. thx
Go to the top of the page
+Quote Post
kjoonlee
post Oct 19 2005, 18:31
Post #4





Group: Members
Posts: 2526
Joined: 25-July 02
From: South Korea
Member No.: 2782



QUOTE (Donunus @ Oct 19 2005, 08:06 PM)
actually can you give me a link that explains placebo. thx
*

The word placebo, when used in audio circles, is a little different from the medical term, where people can actually benefit a little from dummy treatments.

According to ff123, it's closer to expectation bias, where a person's expectation can influence his perception, if I understand correctly.


--------------------
http://blacksun.ivyro.net/vorbis/vorbisfaq.htm
Go to the top of the page
+Quote Post
Woodinville
post Oct 21 2005, 01:27
Post #5





Group: Members
Posts: 1403
Joined: 9-January 05
From: JJ's office.
Member No.: 18957



QUOTE (kjoonlee @ Oct 19 2005, 09:31 AM)
QUOTE (Donunus @ Oct 19 2005, 08:06 PM)
actually can you give me a link that explains placebo. thx
*

The word placebo, when used in audio circles, is a little different from the medical term, where people can actually benefit a little from dummy treatments.

According to ff123, it's closer to expectation bias, where a person's expectation can influence his perception, if I understand correctly.
*



While I don't disagree, I'd go farther.

If you look at what the auditory system detects as an "event", it appears from the work done to be set to detect many 'false events' at a much lower risk of missing a real event.

This kind of performance (tradeoff between misses and false detections) is obligatory for any probabilistic system, of which the auditory system is quite.

I have heard this explained to others (somebody said this at a tutorial at the NY aes, for instance) as being for a simple reason:

If you hear the tiger coming, and hide, you live.

If you don't hear the tiger coming, you are out of the gene pool.

If you hear the tiger coming, and it isn't, all you do is look a bit silly.

If you consider that we are wired to detect nonevents in favor of missing any events, then that would explain quite nicely why changes cause "differences".


--------------------
-----
J. D. (jj) Johnston
Go to the top of the page
+Quote Post
duff
post Oct 21 2005, 01:38
Post #6





Group: Members
Posts: 38
Joined: 18-July 02
From: Los Angeles, CA
Member No.: 2666



QUOTE
If you consider that we are wired to detect nonevents in favor of missing any events, then that would explain quite nicely why changes cause "differences".


One has to be careful about generalizing this principle across all domains. Signal detection applies to many perceptual phenomena, but the costs and benefits associated with the four possible outcomes (hit, miss, false alarm, correct rejection) vary depending on what you are attempting to detect. So we aren't wired to detect "non-events" over events in general. For predator detection, yes...but not for something like, for example, detecting cheaters in a card game!
Go to the top of the page
+Quote Post
Woodinville
post Oct 21 2005, 21:15
Post #7





Group: Members
Posts: 1403
Joined: 9-January 05
From: JJ's office.
Member No.: 18957



QUOTE (duff @ Oct 20 2005, 04:38 PM)
So we aren't wired to detect "non-events" over events in general. For predator detection, yes...but not for something like, for example, detecting cheaters in a card game!
*


Really, now, would you like to show your evidence?

The evidence that partial loudness differences are overdetected has been in for years.

All auditory stimulii start out as a set of partial loudnesses, as expressed as pulse position modulation in the auditory nerves.

If you're right, this would require a complete revision of the entire understanding of how the human auditory system works. Have you this revision prepared yet?

This post has been edited by Woodinville: Oct 21 2005, 21:15


--------------------
-----
J. D. (jj) Johnston
Go to the top of the page
+Quote Post
duff
post Oct 21 2005, 23:43
Post #8





Group: Members
Posts: 38
Joined: 18-July 02
From: Los Angeles, CA
Member No.: 2666



QUOTE (Woodinville @ Oct 21 2005, 08:15 PM)
QUOTE (duff @ Oct 20 2005, 04:38 PM)
So we aren't wired to detect "non-events" over events in general. For predator detection, yes...but not for something like, for example, detecting cheaters in a card game!
*


Really, now, would you like to show your evidence?

The evidence that partial loudness differences are overdetected has been in for years.

All auditory stimulii start out as a set of partial loudnesses, as expressed as pulse position modulation in the auditory nerves.

If you're right, this would require a complete revision of the entire understanding of how the human auditory system works. Have you this revision prepared yet?
*



When I said we aren't "wired" to detect "non-events" in general, I was referring to any phenomena that signal detection theory can be applied. This is a fundamental tenet of the theory; that is, the criterion location depends on the associated costs of different sorts of errors. How that manifests in various auditory contexts is quite variable depending what the task is. Also, it relates to attention and what the sound source is.

I'm not making myself clear if you think the ideas I've presented require a revision of how hearing works. What I am proposing is a result of processing in the auditory cortex, not the ears, and is consistent with what we know about auditory perception. Your account of partial loudnesses doesn't really seem particularly relevant to the issue of whether processing differences between compressed and uncompressed audio could be measured in a way that ABX testing misses.

QUOTE
It doesn't "make up" anything, it detects, in a very lossy and mathematically imperfect fashion, whatever is there. Since that's a lossy fashion, sometimes it imagines that things are there when they aren't, but that's not "making up the missing information" at all.


My example of the missing fundamental phenomenon illustrates nicely how unconscious inference works. There are many other examples as well. Our brains construct a good deal of what we "perceive" based on what is often quite degraded information. For example, your visual experience is radically different than the information on your retina. Perception has a huge filling in component.

QUOTE
People can easily parse sentences with 'degraded' content, such as sentences where 'the the' appears, as well as other typos involving *missing* words or data.
Has it been determined what level of degradation has to be reached before the *time* it takes to parse a sentence is affected? If two tasks take an effectively indistinguishable amount of time for the same person, then one cannot be said to be more 'difficult' than another, can it? The only other measure of 'difficulty' I could imagine would be something like, the number of neurons engaged, or the amount of blood flow involved.


Actually, there is a ton of research on language processing using written sentences, and even something like two "the's" will slow down the system in a statistically significant way. What is often considered a significant amount of time in a cognitive sense is in reality quite fast (e.g., 100 ms. is a big effect), but people generally don't have a conscious sense of these sorts of time issues. Basically it boils down to neural activity. So two different tasks can seem indistinguishable to someone, but have radically different time courses. So yes, this issue is about brain activity, and because the brain is such an energy hog, most roads to easier processing are taken. In terms of the argument David and I are presenting here, the neural processing differences might manifest in long term fatigue effects that could be responsible for some people's discomfort with lossy formats.

One idea that many people seem to be resisting is the notion that very subtle processing differences can result in subtle long term feelings. There seems to be a tendency to overestimate the amount of information that is accessible to consciousness, and as a result, let conscious judgment be the last word on what is actually perceived. But this is definitely not the case as psychophysical researchers (including psychoacoustics people) have known for over a century.

QUOTE
You haven't demonstrated that it's even an *issue* for lossy schemes. Horses before carts, please.

Ans since we seem still to be in the realm of the suppositional, suppose perception is a matter of stimuli crossing sensory and cognitive *thresholds*. It could be that as long as the 'lossy' (lossy only in comparison to the original stimulus) stimulus still crosses the right thresholds, it's all the same to the brain.
This would mean that the 'best' 'lossy' representations' are simply those that are *good enough* -- and they don't require any more 'effort' to process.


I think I've presented a lot of converging evidence that suggests it could very well be an issue. Like I've said, it's a testable idea. David just presented one idea that would speak to it, and we have other ideas as well. You have to find a cart before the horse can pull it.

laugh.gif

The threshold theory is just another version of the idea discussed earlier concerning why it is more labor intensive, so to speak, to perceive a degraded stimulus rather than a less noisy stimulus. David's example of listening to speech in a noisy bar is a good example. But I will throw up some references soon about neural processing and stimulus quality. As one should expect, degraded signals are more difficult to process.
Go to the top of the page
+Quote Post

Posts in this topic
- mirrorsawlljk   What's the problem with double-blind testing?   Oct 19 2005, 03:03
- - bubka   some people can actually detect specific codecs by...   Oct 19 2005, 03:07
- - TheQat   QUOTE (mirrorsawlljk @ Oct 18 2005, 06:03 PM)...   Oct 19 2005, 03:27
|- - singaiya   QUOTE (TheQat @ Oct 18 2005, 06:27 PM)QUOTE (...   Oct 19 2005, 04:31
|- - Yaztromo   QUOTE (TheQat @ Oct 19 2005, 03:27 AM)Edit: H...   Oct 19 2005, 23:05
- - Tahnru   The closest thing I have seen to a legitimate crit...   Oct 19 2005, 04:29
- - Axon   The Audio Critic is notably pro-DBT.   Oct 19 2005, 07:06
|- - PoisonDan   QUOTE (Axon @ Oct 19 2005, 08:06 AM)The Audio...   Oct 19 2005, 12:18
- - onthejazz   Interesting publication. I like it, too bad its no...   Oct 19 2005, 10:03
|- - Donunus   The writer of the letter in stereophile is incorre...   Oct 19 2005, 11:00
|- - Danimal   QUOTE (Donunus @ Oct 19 2005, 05:00 AM)The wr...   Oct 19 2005, 20:20
|- - Donunus   QUOTE (Danimal @ Oct 20 2005, 03:20 AM)QUOTE ...   Oct 20 2005, 01:51
|- - stephanV   QUOTE (Donunus @ Oct 20 2005, 02:51 AM)Well, ...   Oct 20 2005, 08:48
||- - bryant   QUOTE (stephanV @ Oct 19 2005, 11:48 PM)QUOTE...   Oct 20 2005, 18:11
|||- - Pio2001   QUOTE (bryant @ Oct 20 2005, 07:11 PM)To tell...   Oct 20 2005, 20:48
||||- - singaiya   Thanks for the papers, David. The first one is a g...   Oct 20 2005, 22:13
||||- - duff   QUOTE (Pio2001 @ Oct 20 2005, 07:48 PM)QUOTE ...   Oct 20 2005, 22:52
||||- - stephanV   QUOTE (duff @ Oct 20 2005, 11:52 PM)It's ...   Oct 20 2005, 23:11
|||||- - duff   QUOTE Might be... or might be not. You can't c...   Oct 20 2005, 23:36
|||||- - KikeG   QUOTE (duff @ Oct 20 2005, 11:36 PM)I think i...   Oct 21 2005, 00:00
||||||- - rjamorim   QUOTE (KikeG @ Oct 20 2005, 09:00 PM)Also, ps...   Oct 21 2005, 00:15
||||||- - duff   QUOTE (KikeG @ Oct 20 2005, 11:00 PM)QUOTE (d...   Oct 21 2005, 00:29
||||||- - rjamorim   QUOTE (duff @ Oct 20 2005, 09:29 PM)I'm n...   Oct 21 2005, 01:03
||||||- - Mike Giacomelli   QUOTE (duff @ Oct 20 2005, 04:29 PM)The audit...   Oct 21 2005, 05:56
||||||- - KikeG   QUOTE (duff @ Oct 21 2005, 12:29 AM)From the ...   Oct 21 2005, 08:10
|||||- - stephanV   QUOTE (duff @ Oct 21 2005, 12:36 AM)I think i...   Oct 21 2005, 08:34
|||||- - Woodinville   QUOTE (stephanV @ Oct 20 2005, 11:34 PM)Sorry...   Oct 21 2005, 21:19
||||- - krabapple   QUOTE (duff @ Oct 20 2005, 04:52 PM)QUOTE (Pi...   Oct 21 2005, 05:25
||||- - antz   QUOTE (duff @ Oct 20 2005, 10:52 PM)QUOTE (Pi...   Oct 21 2005, 13:39
||||- - Woodinville   QUOTE (antz @ Oct 21 2005, 04:39 AM)Seems to ...   Oct 21 2005, 21:20
|||- - stephanV   QUOTE (bryant @ Oct 20 2005, 07:11 PM)It turn...   Oct 20 2005, 21:43
||- - Donunus   QUOTE (stephanV @ Oct 20 2005, 03:48 PM)QUOTE...   Oct 21 2005, 04:10
||- - stephanV   QUOTE (Donunus @ Oct 21 2005, 05:10 AM)I do n...   Oct 21 2005, 08:43
||- - user   I recall, we have had successful abx tests even he...   Oct 21 2005, 09:20
||- - Pio2001   QUOTE (user @ Oct 21 2005, 10:20 AM)I recall,...   Oct 21 2005, 12:18
|||- - ff123   QUOTE (Pio2001 @ Oct 21 2005, 03:18 AM)QUOTE ...   Oct 21 2005, 17:43
||- - krabapple   QUOTE (user @ Oct 21 2005, 03:20 AM)I recall,...   Oct 21 2005, 21:54
|- - Cartoon   QUOTE (Donunus @ Oct 20 2005, 02:51 AM)I do h...   Nov 12 2005, 13:40
- - KikeG   The "goosebump", emotional factor can be...   Oct 19 2005, 11:54
|- - Donunus   QUOTE (KikeG @ Oct 19 2005, 06:54 PM)The ...   Oct 19 2005, 12:06
|- - kjoonlee   QUOTE (Donunus @ Oct 19 2005, 08:06 PM)actual...   Oct 19 2005, 18:31
||- - Woodinville   QUOTE (kjoonlee @ Oct 19 2005, 09:31 AM)QUOTE...   Oct 21 2005, 01:27
||- - duff   QUOTE If you consider that we are wired to detect ...   Oct 21 2005, 01:38
||- - Woodinville   QUOTE (duff @ Oct 20 2005, 04:38 PM)So we are...   Oct 21 2005, 21:15
||- - duff   QUOTE (Woodinville @ Oct 21 2005, 08:15 PM)QU...   Oct 21 2005, 23:43
||- - Woodinville   QUOTE (duff @ Oct 21 2005, 02:43 PM)When I sa...   Oct 22 2005, 08:00
|- - Lyx   QUOTE (Donunus @ Oct 19 2005, 01:06 PM)QUOTE ...   Oct 20 2005, 06:07
|- - Donunus   QUOTE (Lyx @ Oct 20 2005, 01:07 PM)QUOTE (Don...   Oct 20 2005, 16:51
|- - Lyx   QUOTE (Donunus @ Oct 20 2005, 05:51 PM)QUOTE ...   Oct 20 2005, 21:10
- - KikeG   Well, I think expectation effects in listening tes...   Oct 19 2005, 19:33
|- - krabapple   QUOTE (KikeG @ Oct 19 2005, 10:33 AM)Well, I ...   Oct 19 2005, 20:12
- - krabapple   QUOTE (mirrorsawlljk @ Oct 18 2005, 06:03 PM)...   Oct 19 2005, 20:06
- - Halcyon   There is nothing inherently bad about double blind...   Oct 19 2005, 20:29
- - ff123   Tests that try to distinguish very small effects *...   Oct 19 2005, 20:53
|- - krabapple   QUOTE (ff123 @ Oct 19 2005, 11:53 AM)Tests th...   Oct 19 2005, 23:40
|- - ff123   QUOTE (krabapple @ Oct 19 2005, 02:40 PM)Inte...   Oct 20 2005, 05:38
- - fcmts   There is another problem similar to wine blind tes...   Oct 19 2005, 22:15
- - stephanV   I don't see how that is a problem. The goal of...   Oct 19 2005, 22:22
- - duff   One distinction relevant to this issue is the diff...   Oct 20 2005, 23:00
- - duff   QUOTE The opposite of "blind", I suppose...   Oct 21 2005, 01:27
- - Jun-Dai   I think it's pretty clear that double-blind te...   Oct 21 2005, 07:22
- - KikeG   For helping doing long-term, casual listening-like...   Oct 21 2005, 08:30
- - duff   QUOTE Seems to be a lack of understanding of somet...   Oct 21 2005, 16:48
|- - stephanV   QUOTE (duff @ Oct 21 2005, 05:48 PM)The reaso...   Oct 21 2005, 17:29
||- - duff   QUOTE (stephanV @ Oct 21 2005, 04:29 PM)QUOTE...   Oct 21 2005, 20:06
||- - stephanV   QUOTE (duff @ Oct 21 2005, 09:06 PM)QUOTE Wh...   Oct 21 2005, 20:21
||- - bryant   QUOTE (stephanV @ Oct 21 2005, 11:21 AM)QUOTE...   Oct 21 2005, 22:53
||- - rjamorim   QUOTE (bryant @ Oct 21 2005, 07:53 PM)But I d...   Oct 21 2005, 23:12
||- - stephanV   QUOTE (bryant @ Oct 21 2005, 11:53 PM)So, if ...   Oct 22 2005, 00:00
||- - ChiGung   QUOTE (bryant @ Oct 21 2005, 10:53 PM)What...   Oct 22 2005, 00:25
|||- - bryant   QUOTE (ChiGung @ Oct 21 2005, 03:25 PM)I have...   Oct 22 2005, 23:39
|||- - ChiGung   just thinking -i do get all sorts of weird sensati...   Oct 23 2005, 00:34
||- - Woodinville   QUOTE (bryant @ Oct 21 2005, 01:53 PM)So, if ...   Oct 22 2005, 07:52
||- - bryant   QUOTE (Woodinville @ Oct 21 2005, 10:52 PM)QU...   Oct 22 2005, 23:32
||- - Woodinville   QUOTE (bryant @ Oct 22 2005, 02:32 PM)The onl...   Oct 24 2005, 21:51
|- - Lyx   QUOTE (duff @ Oct 21 2005, 05:48 PM)QUOTE See...   Oct 21 2005, 17:35
|- - krabapple   QUOTE (duff @ Oct 21 2005, 10:48 AM)QUOTE See...   Oct 21 2005, 22:01
- - duff   QUOTE It sounds counterintuitive that my brain wou...   Oct 22 2005, 00:30
|- - ChiGung   QUOTE (duff @ Oct 22 2005, 12:30 AM)Frankly, ...   Oct 22 2005, 00:47
||- - duff   QUOTE (ChiGung @ Oct 21 2005, 11:47 PM)QUOTE ...   Oct 22 2005, 01:14
||- - ChiGung   QUOTE (duff @ Oct 22 2005, 01:14 AM)The reduc...   Oct 22 2005, 01:26
|- - Woodinville   QUOTE (duff @ Oct 21 2005, 03:30 PM)Filtering...   Oct 22 2005, 08:07
- - ChiGung   A simplistic example - a sine wave of exactly 3122...   Oct 22 2005, 00:59
|- - Woodinville   QUOTE (ChiGung @ Oct 21 2005, 03:59 PM)A simp...   Oct 22 2005, 08:08
|- - ChiGung   QUOTE (Woodinville @ Oct 22 2005, 08:08 AM)QU...   Oct 22 2005, 14:09
- - ChiGung   -Sorry for flooding a bit there. Tasty subject   Oct 22 2005, 01:01
- - duff   So sorry for the long delay. I didn't forget a...   Nov 10 2005, 20:10
|- - Woodinville   QUOTE (duff @ Nov 10 2005, 11:10 AM)Woodinsid...   Nov 10 2005, 20:23
- - duff   I don't understand why you would want to maint...   Nov 11 2005, 00:41
|- - Woodinville   QUOTE (duff @ Nov 10 2005, 03:41 PM)Somehow, ...   Nov 11 2005, 21:52
|- - Pio2001   QUOTE (duff @ Nov 11 2005, 01:41 AM)The ABX p...   Nov 12 2005, 03:00
|- - Woodinville   QUOTE (Pio2001 @ Nov 11 2005, 06:00 PM)I can ...   Nov 12 2005, 09:33
|- - user   There aren't theoretical flaws with abx or DB ...   Nov 12 2005, 10:52
- - KikeG   Sorry, you still have provided no evidence that wh...   Nov 11 2005, 14:34
|- - Woodinville   QUOTE (KikeG @ Nov 11 2005, 05:34 AM)Sorry, y...   Nov 11 2005, 21:56
|- - Pio2001   QUOTE (Woodinville @ Nov 11 2005, 10:56 PM)Su...   Nov 12 2005, 02:32
- - duff   Let's back up a moment... There are at least ...   Nov 11 2005, 18:37
- - KikeG   About the first point, your degraded examples are ...   Nov 11 2005, 19:57
- - duff   QUOTE Well, if it's inaudible, there's a g...   Nov 11 2005, 21:19
- - Woodinville   QUOTE (duff @ Nov 11 2005, 12:19 PM)In other ...   Nov 11 2005, 21:59
4 Pages V   1 2 3 > » 


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 23rd November 2014 - 07:03