IPB

Welcome Guest ( Log In | Register )

Overcoming the Perception Problem
item
post Oct 8 2012, 19:21
Post #1





Group: Members
Posts: 33
Joined: 7-August 12
Member No.: 102085



I've noticed an increase in forum debate about the validity of transferring the credibility of ABX from the physical domain to perception testing. I'm wondering if anyone has found a way past this issue?

The purpose of blind testing is to subtract subjectivity from the effect of - for instance - a drug trial: to assess a medication's impact on a subject's physiology with interference from their psychology. But what about when the purpose of a test is subjective perception? How do we then subtract the effect of the method to arrive at a meaningful outcome?

While we would like to remove expectation bias from the equation, if the conditions under which this is done also change the perceptive state of the listener, the test is invalidated as surely as they would be by tissue sample contamination.

Recent large scale public experiments by Lotto Labs (http://www.lottolab.org/) demonstrated that perception acuity is dramatically altered by test conditions: for instance, that time contraction/dilation effects are experienced when exposed to colour fields. In one experiment, two groups were asked to perform an identical fine-grained visual acuity test. One group was pre-emptively 'manipulated' by filling in a questionnaire designed to lower their self-esteem. This 'less confident' group consistently performed worse on the test that the unmanipulated one: their acuity was significantly impaired by a subtle psychological 'tweak' that wasn't even in effect during the test.

It seems undeniable that the much grosser differences between the mental states of sighted and 'blind' listening - considered generously - cast serious doubt on the results thus obtained.

The harder line is that blind perception tests are a fundamental misappropriation of methodology. In psychology it's axiomatic that for many experiments the subject must be unaware of the nature of the test (see Milgram). If a normalised state is not cunningly contrived, results are at best only indicative of what a subject thinks they should do; at worst, entirely invalid.

Probing hearing, the point is that a test must not change the mental state of the listener.

The contrast between outcomes of sighted and listening tests is as stark as those demonstrating suggestibility (see McGurk), but giving too much credence to such an intrinsically unsound experimental approach (not spotting this difficulty) does no favours to our credibility at all.

The only way past the dilemma seems to be direct mechanical examination of the mind during 'normal' listening to explore why the experiences of sighted and unsighted listening differ. This seems to be an interesting question.

In the meantime, the idea that - despite the method problem - results from blind ABX are valid is at least supported by the majority of data derived from home testing, Audio DiffMaker et al, so we needn't get hung up on it.
Go to the top of the page
+Quote Post
 
Start new topic
Replies
greynol
post Oct 9 2012, 04:54
Post #2





Group: Super Moderator
Posts: 10000
Joined: 1-April 04
From: San Francisco
Member No.: 13167



@item:
Perhaps you could share with us a little about who you are so that we can put your point of view into proper perspective.

This site has TOS #8 in place to keep the signal to noise ratio high. As has been aptly pointed out, sighted tests provide absolutely no guarantee of reliability, whereas positive double-blind tests do. The only concern on the table that could possibly have any validity (I am being generous) is that double-blind testing might reduce the sensitivity of the person taking the test. This is perfectly fine since a failed DBT is not used as universal proof that two things must sound the same which is generally where those arguing on behalf of placebophiles get tripped up. FWIW, as a professional tester I can tell you that I actually pay closer attention to detail when I am consciously involved in a test, despite DBT skeptics and snake oil salesmen telling me that I can't or don't.

This post has been edited by greynol: Oct 9 2012, 04:56


--------------------
I should publish a list of forum idiots.
Go to the top of the page
+Quote Post
item
post Oct 9 2012, 11:42
Post #3





Group: Members
Posts: 33
Joined: 7-August 12
Member No.: 102085



QUOTE (greynol @ Oct 9 2012, 04:54) *
@item:
Perhaps you could share with us a little about who you are so that we can put your point of view into proper perspective.


What do you mean by 'proper perspective'? Am I looking at an ad hominem warmup or a chat-up line?

QUOTE (greynol @ Oct 9 2012, 04:54) *
This site has TOS #8 in place to keep the signal to noise ratio high. As has been aptly pointed out, sighted tests provide absolutely no guarantee of reliability, whereas positive double-blind tests do. The only concern on the table that could possibly have any validity (I am being generous) is that double-blind testing might reduce the sensitivity of the person taking the test. This is perfectly fine since a failed DBT is not used as universal proof that two things must sound the same which is generally where those arguing on behalf of placebophiles get tripped up.


Positive DBT is inherently cast-iron. The problem is that negative results equally indict the efficacy of the method, and that DBT perception tests are anathema: they generate results with poor resolution: they conform suspiciously well to the 'bad test' model: ie, they generate positives for gross phenomena but fail to recognise fine-grained distinctions. Wrong sieve size is a plausible diagnosis. Given that the test is misappropriated from a different domain and therefore - by definition - crudely tampers with its objective, this isn't surprising.

QUOTE (greynol @ Oct 9 2012, 04:54) *
FWIW, as a professional tester I can tell you that I actually pay closer attention to detail when I am consciously involved in a test, despite DBT skeptics and snake oil salesmen telling me that I can't or don't.

That's exactly the point: test conditions create an environment in which you have to 'pay closer attention' - in reality, listen in an entirely different way, disorientated and deprived of cues. For a psych test, that's inadmissable.

Again, the purpose of DBT is to remove subjectivity as a factor. It can't legitimately be applied with any degree of precision to a study of subjectivity. Negative DBT results in the physiological domain are always open to question, but in this domain they aren't even interesting, and it's an embarrassment to the cause to see such faith placed in them.

This post has been edited by item: Oct 9 2012, 11:44
Go to the top of the page
+Quote Post
2Bdecided
post Oct 9 2012, 14:15
Post #4


ReplayGain developer


Group: Developer
Posts: 5139
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



QUOTE (item @ Oct 9 2012, 11:42) *
Positive DBT is inherently cast-iron. The problem is that negative results equally indict the efficacy of the method, and that DBT perception tests are anathema: they generate results with poor resolution: they conform suspiciously well to the 'bad test' model: ie, they generate positives for gross phenomena but fail to recognise fine-grained distinctions. Wrong sieve size is a plausible diagnosis. Given that the test is misappropriated from a different domain and therefore - by definition - crudely tampers with its objective, this isn't surprising.
With respect, this is demonstrably wrong. The core tests that probe the very limits of human hearing use blind testing, and deliver results that match predictions from the known physiology of the ear. To get these results takes careful training - people need to learn what to listen for before they can hear as well as the physiology would predict.


You've written many words, but like most blind testing bashing, it comes down to this: "when people are under test, they listen differently so we can't know what they really hear. If they don't know what they are listening to, they are even more stressed." What if they do know what they are listening to, like in most hi-fi magazine reviews? The people are still "under test", yet seem to hear just fine? Given that knowing what you are listening to is both the only differentiating variable, and a known feature that will give completely unreliable results, you are either wrong (that's my guess), or are correct and have just kicked audio into a "never possible to know" philosophical world.

Cheers,
David.
Go to the top of the page
+Quote Post
item
post Oct 9 2012, 14:37
Post #5





Group: Members
Posts: 33
Joined: 7-August 12
Member No.: 102085



QUOTE (2Bdecided @ Oct 9 2012, 14:15) *
QUOTE (item @ Oct 9 2012, 11:42) *
Positive DBT is inherently cast-iron. The problem is that negative results equally indict the efficacy of the method, and that DBT perception tests are anathema: they generate results with poor resolution: they conform suspiciously well to the 'bad test' model: ie, they generate positives for gross phenomena but fail to recognise fine-grained distinctions. Wrong sieve size is a plausible diagnosis. Given that the test is misappropriated from a different domain and therefore - by definition - crudely tampers with its objective, this isn't surprising.
With respect, this is demonstrably wrong. The core tests that probe the very limits of human hearing use blind testing, and deliver results that match predictions from the known physiology of the ear. To get these results takes careful training - people need to learn what to listen for before they can hear as well as the physiology would predict.


You've written many words, but like most blind testing bashing, it comes down to this: "when people are under test, they listen differently so we can't know what they really hear. If they don't know what they are listening to, they are even more stressed." What if they do know what they are listening to, like in most hi-fi magazine reviews? The people are still "under test", yet seem to hear just fine? Given that knowing what you are listening to is both the only differentiating variable, and a known feature that will give completely unreliable results, you are either wrong (that's my guess), or are correct and have just kicked audio into a "never possible to know" philosophical world.

Cheers,
David.

'Knowing what you are listening to' is the troublesome variable. The power of suggestion derives from pre-erecting a preliminary framework with given reference points. The mind obediently attaches incoming sense data to that superstructure because it's hard, slow work to build a model from scratch. Proprioception is another example of the mind slowly building consistent external reality models only via trial and error cycles (Oliver Sacks has a fine description of this in 'A Leg to Stand On').

Deprived of reference points, acuity suffers. Not badly enough to become deaf, obviously - but badly enough to diminish large variables to small ones, and make small ones vanish entirely - which isn't a bad one-line summary of DBT perception results: particularly with reference to hearing which - being driven by feebler mental horsepower - is more prone to suggestion (and more in need of supporting frameworks) than sight (hence McGurk).

Either this is wrong (as you say), or truly accurate testing of this type will come later, when we can directly, mechanically examine - and analyse - brain response without tampering with the subject's psychological state. Certainly not a 'never possible to know' scenario.
Go to the top of the page
+Quote Post
2Bdecided
post Oct 9 2012, 17:43
Post #6


ReplayGain developer


Group: Developer
Posts: 5139
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



QUOTE (item @ Oct 9 2012, 14:37) *
Deprived of reference points, acuity suffers. Not badly enough to become deaf, obviously - but badly enough to diminish large variables to small ones, and make small ones vanish entirely - which isn't a bad one-line summary of DBT perception results: particularly with reference to hearing which - being driven by feebler mental horsepower - is more prone to suggestion (and more in need of supporting frameworks) than sight (hence McGurk).
Are you claiming people are more prone to hearing things that aren't there than seeing things that aren't there - in respect of the brain "filling in gaps" to create what turns out to be an incorrect model of the real world? I'm not sure that's true.


QUOTE
Either this is wrong (as you say), or truly accurate testing of this type will come later, when we can directly, mechanically examine - and analyse - brain response without tampering with the subject's psychological state. Certainly not a 'never possible to know' scenario.
...but they've already hacked cat's heads about to find out what signals went in/out of the auditory nerve long before it was possible to do this in a humane way. It's relevant because they have similar cochlea to us, and it's found that, like us, the losses (what you can't hear) derive from the air to neural transduction process.

So we've got cut up cats, predictions from physiology, and blind tests on humans all delivering the same "what difference is just audible" data - but you don't trust the blind tests?


Yet you'll trust the brain response. That's strange, since no one is doubting that placebo is a real brain response - it's just not a response to what you hear! wink.gif And if we measure some brain response when people are not aware of what they're listening to (i.e. some response to A that is absent to B), either this will be associated with a conscious audible difference, or not. If not, who cares. If so, then having reported hearing it when not knowing what they were listening to, they've just passed a blind test!

What does the brain scan add? Think this through. Draw a flowchart of the possibilities if it helps.

Cheers,
David.
Go to the top of the page
+Quote Post
item
post Oct 19 2012, 17:34
Post #7





Group: Members
Posts: 33
Joined: 7-August 12
Member No.: 102085



Sorry - been away, but lots of noise (most - fascinatingly - irrelevant) generated in the interim but, to respond to a few points . . .

QUOTE (2Bdecided @ Oct 9 2012, 17:43) *
Are you claiming people are more prone to hearing things that aren't there than seeing things that aren't there - in respect of the brain "filling in gaps" to create what turns out to be an incorrect model of the real world? I'm not sure that's true.


Not what I said, or McGurk shows. Please re-read.

QUOTE (2Bdecided @ Oct 9 2012, 17:43) *
...but they've already hacked cat's heads about to find out what signals went in/out of the auditory nerve long before it was possible to do this in a humane way. It's relevant because they have similar cochlea to us, and it's found that, like us, the losses (what you can't hear) derive from the air to neural transduction process. So we've got cut up cats, predictions from physiology, and blind tests on humans all delivering the same "what difference is just audible" data - but you don't trust the blind tests? Yet you'll trust the brain response. That's strange, since no one is doubting that placebo is a real brain response - it's just not a response to what you hear! wink.gif And if we measure some brain response when people are not aware of what they're listening to (i.e. some response to A that is absent to B), either this will be associated with a conscious audible difference, or not. If not, who cares. If so, then having reported hearing it when not knowing what they were listening to, they've just passed a blind test! What does the brain scan add? Think this through. Draw a flowchart of the possibilities if it helps.
Cheers,
David.

We don't report a fraction of our brain response. Humans are not cats. And placebo is not relevant to either.
Go to the top of the page
+Quote Post

Posts in this topic
- item   Overcoming the Perception Problem   Oct 8 2012, 19:21
- - Soap   You appear to be confused. Despite common shortha...   Oct 8 2012, 19:38
- - item   To rephrase: what, in any trial, is blind testing ...   Oct 8 2012, 19:52
|- - Soap   QUOTE (item @ Oct 8 2012, 14:52) To rephr...   Oct 8 2012, 19:59
|- - item   QUOTE (Soap @ Oct 8 2012, 19:59) QUOTE (i...   Oct 8 2012, 23:32
|- - item   Part of Beau Lotto's 'Public Perception pr...   Oct 9 2012, 00:30
||- - Porcus   QUOTE (item @ Oct 9 2012, 01:30) Part of ...   Oct 9 2012, 10:53
||- - item   QUOTE (Porcus @ Oct 9 2012, 10:53) You ca...   Oct 9 2012, 12:23
||- - Porcus   QUOTE (item @ Oct 9 2012, 13:23) This for...   Oct 9 2012, 16:04
|- - krabapple   QUOTE (item @ Oct 8 2012, 18:32) Partly, ...   Oct 9 2012, 03:51
|- - item   QUOTE (item @ Oct 8 2012, 18:32) Although...   Oct 9 2012, 11:14
- - DVDdoug   I just don't see how making a good scientific ...   Oct 9 2012, 00:59
- - greynol   @item: Perhaps you could share with us a little ab...   Oct 9 2012, 04:54
|- - Porcus   QUOTE (greynol @ Oct 9 2012, 05:54) a fai...   Oct 9 2012, 09:57
|- - item   QUOTE (greynol @ Oct 9 2012, 04:54) @item...   Oct 9 2012, 11:42
|- - 2Bdecided   QUOTE (item @ Oct 9 2012, 11:42) Positive...   Oct 9 2012, 14:15
||- - item   QUOTE (2Bdecided @ Oct 9 2012, 14:15) QUO...   Oct 9 2012, 14:37
||- - 2Bdecided   QUOTE (item @ Oct 9 2012, 14:37) Deprived...   Oct 9 2012, 17:43
|||- - item   Sorry - been away, but lots of noise (most - fasci...   Oct 19 2012, 17:34
||- - dhromed   QUOTE (item @ Oct 9 2012, 15:37) 'Kno...   Oct 9 2012, 19:11
|- - greynol   QUOTE (item @ Oct 9 2012, 03:42) QUOTE (g...   Oct 9 2012, 22:07
|- - item   QUOTE (greynol @ Oct 9 2012, 22:07) @item...   Oct 19 2012, 18:33
- - hlloyge   The only people I know of to shun DBT method of te...   Oct 9 2012, 12:35
|- - item   QUOTE (hlloyge @ Oct 9 2012, 12:35) The o...   Oct 9 2012, 13:25
|- - hlloyge   QUOTE (item @ Oct 9 2012, 14:25) The sole...   Oct 9 2012, 13:54
||- - item   QUOTE (hlloyge @ Oct 9 2012, 13:54) The o...   Oct 9 2012, 14:09
|- - Canar   QUOTE (item @ Oct 9 2012, 05:25) I think ...   Oct 9 2012, 17:09
|- - krabapple   QUOTE (item @ Oct 9 2012, 08:25) The sole...   Oct 10 2012, 01:25
|- - item   QUOTE (krabapple @ Oct 10 2012, 01:25) QU...   Oct 19 2012, 18:45
|- - Porcus   QUOTE (item @ Oct 19 2012, 19:45) QUOTE (...   Oct 19 2012, 19:22
|- - 2Bdecided   QUOTE (item @ Oct 19 2012, 18:45) Certain...   Oct 22 2012, 12:53
- - aethelberht   What I can perhaps maybe possibly gather from your...   Oct 9 2012, 13:17
|- - item   QUOTE (aethelberht @ Oct 9 2012, 13:17) W...   Oct 9 2012, 14:01
|- - aethelberht   "A negative means - equally - either a) the t...   Oct 9 2012, 14:14
|- - Arnold B. Krueger   QUOTE (item @ Oct 9 2012, 09:01) DBT is d...   Oct 9 2012, 14:53
|- - hlloyge   QUOTE (item @ Oct 9 2012, 15:01) Abstract...   Oct 10 2012, 21:04
|- - Porcus   QUOTE (hlloyge @ Oct 10 2012, 22:04) If y...   Oct 11 2012, 14:20
|- - hlloyge   QUOTE (Porcus @ Oct 11 2012, 15:20) Ehem ...   Oct 12 2012, 14:26
|- - Porcus   QUOTE (hlloyge @ Oct 12 2012, 15:26) QUOT...   Oct 12 2012, 15:04
|- - hlloyge   QUOTE (Porcus @ Oct 12 2012, 16:04) Not n...   Oct 12 2012, 22:14
|- - greynol   QUOTE (hlloyge @ Oct 12 2012, 14:14) I am...   Oct 13 2012, 14:25
|- - hlloyge   QUOTE (greynol @ Oct 13 2012, 15:25) QUOT...   Oct 14 2012, 13:35
|- - Porcus   QUOTE (hlloyge @ Oct 14 2012, 14:35) Yes,...   Oct 14 2012, 23:31
|- - hlloyge   QUOTE (Porcus @ Oct 15 2012, 00:31) QUOTE...   Oct 16 2012, 09:54
- - Porcus   While I certainly agree that putting humans in a l...   Oct 10 2012, 13:09
- - 2Bdecided   It's like The Princess and the Pea. Audiophile...   Oct 11 2012, 11:58
- - skamp   If ABXing negatively alters one's ability to h...   Oct 11 2012, 14:28
|- - krabapple   QUOTE (skamp @ Oct 11 2012, 09:28) If ABX...   Oct 11 2012, 18:24
|- - mzil   QUOTE (krabapple @ Oct 11 2012, 13:24) QU...   Oct 15 2012, 17:14
|- - krabapple   You're right that different terms apply when w...   Oct 15 2012, 20:51
- - greynol   I believe our skeptic has flown the coop.   Oct 11 2012, 16:17
- - dhromed   Not everyone is as much a netizen as most of us. P...   Oct 11 2012, 17:05
- - googlebot   While the OP's reasoning and claimed inference...   Oct 11 2012, 21:44
|- - [JAZ]   QUOTE (googlebot @ Oct 11 2012, 22:44) Do...   Oct 11 2012, 22:37
||- - bandpass   QUOTE ([JAZ] @ Oct 11 2012, 22:37...   Oct 12 2012, 06:44
|- - sld   QUOTE (googlebot @ Oct 12 2012, 04:44) Lo...   Oct 12 2012, 03:38
|- - krabapple   QUOTE (googlebot @ Oct 11 2012, 16:44) Wh...   Oct 12 2012, 06:59
||- - googlebot   QUOTE (krabapple @ Oct 12 2012, 07:59) Ex...   Oct 12 2012, 21:44
||- - Nick.C   QUOTE (googlebot @ Oct 12 2012, 21:44) Th...   Oct 12 2012, 21:51
|||- - googlebot   QUOTE (Nick.C @ Oct 12 2012, 22:51) Is th...   Oct 12 2012, 21:58
|||- - Nick.C   QUOTE (googlebot @ Oct 12 2012, 21:58) Ye...   Oct 12 2012, 22:21
|||- - googlebot   I do not see how calling the phenomenon "prec...   Oct 12 2012, 23:43
||- - krabapple   QUOTE (googlebot @ Oct 12 2012, 16:44) QU...   Oct 13 2012, 22:05
|- - 2Bdecided   @Porcus: QUOTE (googlebot @ Oct 11 2012, 21...   Oct 12 2012, 10:08
|- - Arnold B. Krueger   QUOTE (googlebot @ Oct 11 2012, 16:44) Co...   Oct 16 2012, 15:51
- - greynol   I would be careful not to limit the word perceive....   Oct 11 2012, 22:58
- - greynol   I extract all the joy I could ever need from simpl...   Oct 12 2012, 06:11
- - Porcus   Among my friends, we have been blind testing ... h...   Oct 12 2012, 09:30
- - dhromed   But is there a problem?   Oct 12 2012, 22:12
- - AndyH-ha   The sighted test difference is not coming from the...   Oct 13 2012, 03:32
|- - greynol   QUOTE (AndyH-ha @ Oct 12 2012, 19:32...   Oct 13 2012, 14:16
- - Nick.C   @googlebot: You are now allowing the results to be...   Oct 13 2012, 09:43
- - 2Bdecided   I think Googlebot is making a valid philosophical ...   Oct 15 2012, 12:20
|- - pisymbol   QUOTE (2Bdecided @ Oct 15 2012, 07:20) I ...   Oct 15 2012, 13:48
- - skamp   QUOTE (mzil @ Oct 15 2012, 18:14) B. The ...   Oct 15 2012, 22:49
|- - krabapple   QUOTE (skamp @ Oct 15 2012, 17:49) What g...   Oct 16 2012, 03:58
- - greynol   There are ways of cheating to get positive ABX res...   Oct 15 2012, 22:57
- - mzil   [Trying to bring this back on topic] There is not...   Oct 16 2012, 04:32
|- - Woodinville   QUOTE (mzil @ Oct 15 2012, 20:32) [Trying...   Oct 16 2012, 04:33
|- - Porcus   QUOTE (mzil @ Oct 16 2012, 05:32) Here...   Oct 16 2012, 06:19
- - mzil   Please enlighten me. I am not a scientist nor have...   Oct 16 2012, 04:39
|- - krabapple   QUOTE (mzil @ Oct 15 2012, 23:39) Please ...   Oct 16 2012, 22:28
|- - mzil   QUOTE (krabapple @ Oct 16 2012, 17:28) QU...   Oct 17 2012, 00:30
|- - saratoga   QUOTE (mzil @ Oct 16 2012, 19:30) So even...   Oct 17 2012, 03:03
||- - mzil   QUOTE (saratoga @ Oct 16 2012, 22:03) QUO...   Oct 17 2012, 05:35
||- - saratoga   QUOTE (mzil @ Oct 17 2012, 00:35) Sorry, ...   Oct 17 2012, 05:43
|- - krabapple   QUOTE (mzil @ Oct 16 2012, 19:30) QUOTE (...   Oct 17 2012, 04:53
||- - mzil   QUOTE (krabapple @ Oct 16 2012, 23:53) QU...   Oct 17 2012, 06:36
||- - mzil   QUOTE (krabapple @ Oct 16 2012, 23:53) Do...   Oct 17 2012, 07:22
|||- - mzil   QUOTE (mzil @ Oct 17 2012, 02:22) QUOTE (...   Oct 18 2012, 19:16
||- - Arnold B. Krueger   QUOTE (krabapple @ Oct 16 2012, 23:53) Do...   Oct 18 2012, 14:04
|- - Porcus   QUOTE (mzil @ Oct 17 2012, 01:30) QUOTE R...   Oct 17 2012, 18:51
- - knutinh   Self-reporting about ones mental state surely carr...   Oct 16 2012, 21:02
- - greynol   Let me get this straight, the subconscious mind is...   Oct 17 2012, 05:28
- - mzil   Thanks, Porcus. I'll check it out.   Oct 17 2012, 20:14
- - krabapple   It's best to be careful drawing conclusions fr...   Oct 19 2012, 17:53
|- - 2Bdecided   QUOTE (krabapple @ Oct 19 2012, 17:53) It...   Oct 22 2012, 12:49
- - AndyH-ha   I could be missing the point, or dozens of them, b...   Oct 19 2012, 20:56
|- - Arnold B. Krueger   QUOTE (AndyH-ha @ Oct 19 2012, 15:56...   Oct 19 2012, 21:30
- - AndyH-ha   My point was not that many reported tests involve ...   Oct 19 2012, 22:31
2 Pages V   1 2 >


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 22nd September 2014 - 08:53