Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: What's the problem with double-blind testing? (Read 245619 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

What's the problem with double-blind testing?

Reply #125 –
My perception of X as A and not B implies that I believe I can differentiate either X from Y or A from B since by definition X either equals A or B, and consequently Y equals whatever X does not (assuming that A does not equal B).  Regardless of what your skeptic thinks, he has no credible argument.

Semantics aside, if the "experience" between A and B is truly different to the experiencer, then there is no reason it can't be measured by a double-blind test.

What's the problem with double-blind testing?

Reply #126 –
You have completely missed the point of ABX. You are presented with X which is the same as either A or B, and is different from the other one! The latter statement is the same as your statement #2, but to prove that you can hear a difference between A and B you must choose which one it sounds different from, A or B.

Edit: greynol beat me to it.

What's the problem with double-blind testing?

Reply #127 –
Quote from: greynol on
My perception of X as A and not B implies that I believe I can differentiate either X from Y or A from B since by definition X either equals A or B, and consequently Y equals whatever X does not (assuming that A does not equal B).  Regardless of what your skeptic thinks, he has no credible argument.

Semantics aside, if the "experience" between A and B is truly different to the experiencer, then there is no reason it can't be measured by a double-blind test.


That may be your definition of "perceives X as A and not B" (to change the variables in the original example), but there is no necessity for the skeptic to accept that definition.  Why does perceiving necessarily imply anything about having beliefs about what you can differentiate?  Animals perceive, but they likely do not have any beliefs about what they can differentiate.

What's the problem with double-blind testing?

Reply #128 –
Quote from: pdq on
You have completely missed the point of ABX. You are presented with X which is the same as either A or B, and is different from the other one! The latter statement is the same as your statement #2, but to prove that you can hear a difference between A and B you must choose which one it sounds different from, A or B.

Edit: greynol beat me to it.


No, I was saying that the person in the example acknowledges that he can't hear a difference.  Please look again at my post.

What's the problem with double-blind testing?

Reply #129 –
What part about semantics aside do you not comprehend?

Perceive a difference can be the result of anything.  It could be a feeling in your stomach or a taste in your mouth.

The ability to perceive something as one thing and not the other means you can differentiate the two.

If the skeptic is not interested in understanding how ABX works, why should anyone care what he has to say about it?

What's the problem with double-blind testing?

Reply #130 –
I think that ending up essentially arguing about what consciousness is in order to find something wrong with ABX-type tests is pretty funny.

But maybe I am too literal-minded.

Anyway, no offense meant to anybody.

What's the problem with double-blind testing?

Reply #131 –
Look, I just did an ABX of the same piece at different sampling rates and I couldn't hear a difference.  And I asked, What exactly does this tell me?  Does it tell me that those files don't differ in properties that, when I listen in 10-minute stretches, cause me to have different types of experiences?  (Where this difference is one that matters: where I have an interest in having experiences of one type over the other.)

As far as I can see, it does not tell me that, and I have never seen a convincing argument that it does.

What's the problem with double-blind testing?

Reply #132 –
It certainly doesn't demonstrate that you actually perceive a difference, either.  If you think you can do better beyond 10 minute stretches, feel free.

Personally I think you're just grasping at straws in an effort to not accept the fact that you are merely deluding yourself.

Either way, you have failed in your attempt to discredit ABX as a tool to identify one's ability to perceive/distinguish/whateveryouwannacallit differences.

What's the problem with double-blind testing?

Reply #133 –
Quote from: aclo on
I think that ending up essentially arguing about what consciousness is in order to find something wrong with ABX-type tests is pretty funny.

But maybe I am too literal-minded.

Anyway, no offense meant to anybody.


None taken.  FWIW, I wasn't saying that there was anything "wrong" with ABX tests.  The question is what they do or do not prove.  One thing that raised my eyebrows is the assertion way back in this thread that negative ABX results "debunk" certain claims.  I suppose it depends on what one means by "debunk" and for what purposes.  The question here is, do the files in my example differ in their properties in a way that causes you to hear different things, when you listen normally for long stretches of time?  I have never seen a convincing argument as to why a negative ABX test settles that question.

What's the problem with double-blind testing?

Reply #134 –
Quote from: greynol on
It certainly doesn't demonstrate that you actually perceive a difference, either.  If you think you can do better beyond 10 minute stretches, feel free.

Personally I think you're just grasping at straws in an effort to not accept the fact that you are merely deluding yourself.

Either way, you have failed in your attempt to discredit ABX as a tool to identify one's ability to perceive/distinguish/whateveryouwannacallit differences.


Fine, but please note that that isn't what I was attempting to do in the first place, so I don't think you understood what I was saying.

What's the problem with double-blind testing?

Reply #135 –
Quote from: greynol on
What part about semantics aside do you not comprehend?

Perceive a difference can be the result of anything.  It could be a feeling in your stomach or a taste in your mouth.

The ability to perceive something as one thing and not the other means you can differentiate the two.

If the skeptic is not interested in understanding how ABX works, why should anyone care what he has to say about it?


RE: "The ability to perceive something as one thing and not the other means you can differentiate the two."

Says you.  I walk into a museum and something looks like a Picasso to me.  Does that mean that I can (reliably) differentiate Picassos from non-Picassos? 

RE: "If the skeptic is not interested in understanding how ABX works, why should anyone care what he has to say about it?"

I don't know; who said the skeptic wasn't interested in that?

What's the problem with double-blind testing?

Reply #136 –
Quote from: Mark DeB on
Quote from: aclo on
I think that ending up essentially arguing about what consciousness is in order to find something wrong with ABX-type tests is pretty funny.

But maybe I am too literal-minded.

Anyway, no offense meant to anybody.


None taken.  FWIW, I wasn't saying that there was anything "wrong" with ABX tests.  The question is what they do or do not prove.  One thing that raised my eyebrows is the assertion way back in this thread that negative ABX results "debunk" certain claims.  I suppose it depends on what one means by "debunk" and for what purposes.  The question here is, do the files in my example differ in their properties in a way that causes you to hear different things, when you listen normally for long stretches of time?  I have never seen a convincing argument as to why a negative ABX test settles that question.


Well, induction can't actually prove anything anyway. Obviously an a priori argument that results obtained by performing double-blind (for instance) tests on 1-minute pieces apply also to listening to a whole symphony is impossible to construct; just how much generalisation one is willing to allow will depend on one's assumptions and model of what is going on. Using "my model" of what is occurring here, it seems obvious that if something can't be ABXed using eg 15s of music then I won't be able to ABX it with longer pieces either. Clearly you have a different underlying model.

At any rate, one can ABX over longer periods as well.

Anyway, I think both "camps" are applying common sense, and the only difference is what their common sense tells them

What's the problem with double-blind testing?

Reply #137 –
ABX is about whether you can notice the difference between two similar stimuli. It does not prove if two stimuli are the same. An ABX test determines the probability that you actually can make out the difference between the two. However, if the same ABX test is being done by a larger user group, certain assumptions can be made. It's all about statistics.

Quote
Anyway, I think both "camps" are applying common sense, and the only difference is what their common sense tells them

No, one camp uses scientific evidence within a certain, small range of uncertainty while the other camp is zealous about it's unfounded belief.

What's the problem with double-blind testing?

Reply #138 –
Quote from: Mark DeB on
I have never seen a convincing argument as to why a negative ABX test settles that question.

Individual ABX tests that are negative don't prove the impossibility that someone will never perceive a difference.  This has never been disputed by those who understand the methodology.  It's hardly any basis to demonstrate that you can perceive a difference, however.

What's the problem with double-blind testing?

Reply #139 –
Quote from: Mark DeB on
Says you.  I walk into a museum and something looks like a Picasso to me.  Does that mean that I can (reliably) differentiate Picassos from non-Picassos?
Non sequitur.  I never suggested that one had to identify the precise origin of a sample under test, nor are we comparing apples to oranges, either.

What's the problem with double-blind testing?

Reply #140 –
Mark DeB, your point just doesn't hold. You claim there might be an longterm benefit to A that B hasn't and that the difference between A/B might not be noticeable by direct (short term) comparison. As others have said, nothing prevents you from extending test exposure to longer periods, say a whole day. After, for example, a month, you should have marked several days in your calendar with "experienced effect" and some with "no effect" then compare the result with the actual settings.

The strongest effect I have experienced so far is sight. The effect is so strongly perceivable, especially when comparing audio stuff, that it's almost funny. I have to bind myself to blind protocol to not be fooled.

The effect, which you believe to be possibly at work here, is very improbable. Nevertheless it would be verifiable with extended testing. I also would not call your position truly sceptic, rather the opposite. You seem to just resort to a ultrasceptic position to sustain a non-sceptic belief (possibly induced by sighted comparison).

Not being able to perceive a difference between two things means there is no perceivable difference for that subject. The domain of reality outside that scope of that subject's perception has nothing to do with it.

What's the problem with double-blind testing?

Reply #141 –
Quote from: LaserSokrates on
ABX is about whether you can notice the difference between two similar stimuli. It does not prove if two stimuli are the same. An ABX test determines the probability that you actually can make out the difference between the two. However, if the same ABX test is being done by a larger user group, certain assumptions can be made. It's all about statistics.

That's obviously a true statement, but I am not sure why this was posted as a reply to my comment. Unless you're trying to say that one can increase the probability of something being different by having an arbitrarily large group take part in the test, which is also true but also irrelevant.

Quote
Quote
Anyway, I think both "camps" are applying common sense, and the only difference is what their common sense tells them

No, one camp uses scientific evidence within a certain, small range of uncertainty while the other camp is zealous about it's unfounded belief.


Actually it is literally true that both camps (here, I am not talking about magazine editors lying through their teeth for instance) are in fact using their common sense, and it is just that it's telling them different things. I do agree that it's much more reasonable to assume that results obtained by using samples of (for instance) 20s apply also to general music listening than not to, but that's because my commons sense tells me so (ie because I can "watch" myself listening to music and it seems that no integration occurs over such timescales, while also I've observed myself get fooled  by my hearing often enough to not trust subtle results indicating counterintuitive effects--the results would have to be compelling).

Anyway, you can't actually prove it except by induction, ie performing the experiments and showing that the same conclusion is reached. Given that I consider this generalisability to be obviously true, I won't bother  Or you could show that no mechanism exists that would result in long-term listening revealing features that cannot be noticed on short-term listening, but I don't think such an approach would work in this type of psychoacoustical research.

All I am saying is that in this particular case, it seems that Mark DeB is not doing this to defend his irrational beliefs but rather seems to actually be wondering about it. And if you think this is being stupid, that's fine, but I don't. I disagree with what he believes and think it's obviously incorrect, but that's all.

What's the problem with double-blind testing?

Reply #142 –
Quote from: greynol on
Quote from: Mark DeB on
Says you.  I walk into a museum and something looks like a Picasso to me.  Does that mean that I can (reliably) differentiate Picassos from non-Picassos?
Non sequitur.  I never suggested that one had to identify the precise origin of a sample under test, nor are we comparing apples to oranges, either.


No, but it is a counterexample to the principle you stated, since we do sometimes perceive paintings as Picassos.  If you want to make your principle more precise to rule out this counterexample, please feel free.

Look, you are saying (yes?) that from the fact that I can't perceive a difference between two stimuli in a quick-switch ABX test that I should conclude that those stimuli, when I listen in long stretches, don't cause different experiences in me?  Why does that follow, on the basis of either logic or science?  How do I know, for example, that when I listen in long stretches I don't have a different perceptual set from the one I have in the testing situation, and hear different things in response to the stimuli, but lack the ability to compare the experiences because they're too far separated in time?

What's the problem with double-blind testing?

Reply #143 –
Quote from: Mark DeB on
lack the ability to compare the experiences because they're too far separated in time?

You don't lack the ability to compare the experiences because they're too far separated in time.

It seems like every time someone comes along with silly excuses for their inability to demonstrate they aren't simply imagining differences, the conversation devolves into nitpicking over comments that have nothing to do with the principles being discussed.

If you think you can fit your absurd Picasso example into the workings of ABX testing, feel free.  Otherwise, you're wasting our time.


What's the problem with double-blind testing?

Reply #144 –
Quote from: rpp3po on
Mark DeB, your point just doesn't hold. You claim there might be an longterm benefit to A that B hasn't and that the difference between A/B might not be noticeable by direct (short term) comparison. As others have said, nothing prevents you from extending test exposure to longer periods, say a whole day. After, for example, a month, you should have marked several days in your calendar with "experienced effect" and some with "no effect" then compare the result with the actual settings.

The strongest effect I have experienced so far is sight. The effect is so strongly perceivable, especially when comparing audio stuff, that it's almost funny. I have to bind myself to blind protocol to not be fooled.

The effect, which you believe to be possibly at work here, is very improbable. Nevertheless it would be verifiable with extended testing. I also would not call your position truly sceptic, rather the opposite. You seem to just resort to a ultrasceptic position to sustain a non-sceptic belief (possibly induced by sighted comparison).

Not being able to perceive a difference between two things means there is no perceivable difference for that subject. The domain of reality outside that scope of that subject's perception has nothing to do with it.


Right, no perceivable difference, but that isn't the same as no difference between perceptions.  If you listen to a one-minute stretch of music, and then a similar one-minute stretch, you might very well perceive different things, but not be able to detect and/or report this reliably because you  can't compare them.  And there is no need to assume that experience implies the ability to apply a label (such as "experienced effect").

What's the problem with double-blind testing?

Reply #146 –
Quote from: greynol on
Quote from: Mark DeB on
lack the ability to compare the experiences because they're too far separated in time?

You don't lack the ability to compare the experiences because they're too far separated in time.

It seems like every time someone comes along with silly excuses for their inability to demonstrate they aren't simply imagining differences, the conversation devolves into nitpicking over comments that have nothing to do with the principles being discussed.

If you think you can fit your absurd Picasso example into the workings of ABX testing, feel free.  Otherwise, you're wasting our time.


The separation in time might or might not be the salient reason.  But in any case, the question stands: why should I conclude, from what I can or cannot do in the quick-switch ABX situation, anything about whether the stimuli cause different experiences in the long-term setting? And I see nothing in your posts that actually gives a reason why I should conclude that.

What's the problem with double-blind testing?

Reply #148 –
Settle down, chil'ren.

Mark, you are correct that under any strictly logical analysis, a negative ABX result cannot "debunk" the existence or non-existence of a difference. Nonexistence is nonfalsifiable. However, it can disprove the magnitude of such a difference, if it exists, and often times that's enough to settle a debate. If one says "OMG this is a night and day difference, I can detect the improvement in half a second", but then undergoes an ABX test under identical listening conditions and fails it miserably, the most plausible reason for the failure is not that there's some fundamental issue with ABX testing that fundamentally compromises its meaning (although that excuse is profferred all the time). The most plausible reason is that the originally perceived difference was 100% imaginary, and among many reasons why this is the most plausible, the one I prefer is that I observe that behavior in myself, regularly, in comparisons which plainly null out, and based on my experience and what I know of human psychology, I expect that behavior in everybody else.

However, when blind tests consistently fail, and there are good theoretical reasons to doubt that such tested differences would ever be audible, at some point, you've gotta step back and call a turd a turd. A while ago there was a big controlled study showing that homeopathy did not demonstrate any efficacy, and while that was used by doctors to "prove" that homeopathy was a worthless fraud, it clearly could not justify such a conclusion - and it didn't, at least not on its own. But combined with the knowledge that homeopathy treatments are diluted to the point of not containing a single molecule of the original substance, and the knowledge that no remotely scientifically plausible theory exists that would in any way justify its efficacy, even after over 100 years of study, you'd need some pretty thick blinders (or big cahones) to assert anything different than "homeopathy is dead".

In other words... bone up on your Kuhn. Paradigms have never been overturned because of evidence.

Regarding quick-switch, I might offer (anybody correct me if I'm wrong here): a) No evidence exists that long-time switching is any more sensitive than quick switching; b) Considerable anecdotal evidence (and I want to say at least one study) indicates quick switching being more sensitive than long time switching; c) The seconds-long duration of auditory memory provides an extremely powerful theoretical justification for the increased sensitivity of fast switching. There is simply no evidence supporting such a theoretical objection, some (perhaps a significant) amoung of evidence against the idea, and a very good theoretical reason against the idea.

What's the problem with double-blind testing?

Reply #149 –
Why the Picasso's analogy. I've been recently convinced that people who favor pseudoscience are astoundingly bad at analogies. "Perceiving" a painting as a Picasso would be like perceiving a piece as Mozart. Never mind that you're, you know, not even comparing the same piece!

How about this, forget who painted it, and compare the original with an extremely good forgery. A blind test should tell you if you got the chops to spot the differences.