IPB

Welcome Guest ( Log In | Register )

DBT Is Flawed Because Bob Stuart Says So, Split from Topic ID #11442
2Bdecided
post Aug 6 2012, 11:22
Post #1


ReplayGain developer


Group: Developer
Posts: 5360
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



I don't think it's helpful criticising someone for not running proper DBTs of loudspeakers. As far as I know, only one or two members here have tried.

I know the loudspeaker comments were part of a longer post that had many other issues, but I don't think ABX-ing loudspeakers is a realistic thing to jump on.

Cheers,
David.
Go to the top of the page
+Quote Post
 
Start new topic
Replies
audioclaudio
post Aug 8 2012, 20:17
Post #2





Group: Members
Posts: 33
Joined: 3-December 11
Member No.: 95560



The trouble with ABX testing is always the fact if you want it to be truly objective you have to collect a large enough number of subjective test results to start with, and then apply statistics in order to filter out the unwanted subjectiveness part. This is because every individual test result always has a significant chance of being highly inaccurate, so you need lots of people to participate if you want the final conclusions to be reliable ones.
Often, if not practically always, it is too time consuming and/or too uneconomical to conduct an ABX test properly, i.e. in such way that expectation bias doesn't creep in through the back door etcetera. Experts in auditory neuroscience and psychoacoustics have gathered experimental evidence which appears to indicate the following. Humans who remember different things are perfectly capable of hearing the same sounds differently as a result of remembering different things. For example, Bob Stuart of Meridian, who has a Ph.D in neuroscience, believes that it is perfectly possible for a person to not hear a specific detail in a piece of music when it is played back on one particular system "A", then to discover this specific detail by listening to the same music again on a better, more resolving system "B" next, and then, finally, to turn back to the previous system "A" and always hear this detail on system "A" even though the detail could previously not be heard on system "A". Moreover, Bob Stuart believes rapid switching between sounds can inevitably cause humans to perceive sound objects differently.
Go to the top of the page
+Quote Post
[JAZ]
post Aug 11 2012, 16:20
Post #3





Group: Members
Posts: 1797
Joined: 24-June 02
From: Catalunya(Spain)
Member No.: 2383



audioclaudio and Arnold : These two last pages of the thread are really difficult to follow, and don't really help at all to make a decision that answers the topic's title.


First of all, I would like to rewind and set a straight and concrete meaning of several words, since it's not really clear what you're against:

- Objective test/Objectivism ( http://en.wikipedia.org/wiki/Objectivism . In this thread, really http://en.wikipedia.org/wiki/Objectivity_%28science%29 ) : The main idea behind this is that the person doing the test does not matter, since the reality exists independently of the subject taking the test (1 meter is 1 meter no matter who checks it) . Objectivity does make possible to repeat tests and *verify* a result.

- Subjective test/Subjectivism ( http://en.wikipedia.org/wiki/Subjectivism ) : The main idea under subjectivism is that we know the world thanks to our sensory system, and as such, only what we can perceive is what exists (ok. This is just one interpretation. Subjectivism has many variants, from phylosophical, to theological).


- ABX Test: An Objective test that evaluates the response (subjective or not) of the participants to the items at test, and obtains a (statistically) representable result that can be considered reliable and, as such, valid for the specific group at which the study was directed.

- ABX Test, application: An ABX test can only say that a difference exists between the two items being evaluated by the determined subgroup at which the study is directed. (Concretely, this means that it can never be used to say that a difference does not exist. At much, with many failed tests, it can be an indication that such difference is not probable).

In case of Hydrogenaudio, this means that it allows us to *verify* that an *audible difference* between two audio samples (being them two different codecs, two soundcards, two amplifiers....) does exist for the person or people involved in doing the ABX test. If enough tests validate this, then, and only then, can be considered that the difference is real and not due to some error in the test.


Hydrogenaudio is an Objectivism Forum. Let's put that straight and undeniably clear. The high emphasis on ABX tests, the relation with the science world, and the participation of people directly related to the subject of this forum (like the actual coders of audio codecs) is an indication of it.



Reasons why ABX tests are a requirement in order to get objective tests, reliable and representable for the intended audience:


A) Expectation bias ( http://en.wikipedia.org/wiki/Expectation_bias ): A subject, intentionally or not, can be influenced by its subconsciusness: Someone listening to a song in a newly bought equipment does have expectation bias when evaluating its sound quality against an older, less good equipment. (while a difference might exists, the subject could be perceiving his feeling that a difference exists and not the real difference).

B) Representative: To pass an ABX test, it is required to correctly identify a percentage of the trials, and they cannot be cherry picked. ABX pretends to reduce the possibilities of: chance/luck, hidden elements at test (i.e. a difference caused by something that is not the item at test), unconscious influence from other subjects (that's the requirement for the test to be *double* blind).


Setting up an audio ABX test is not necesarily easy, Here are some examples of what can go wrong:

- Playing at different sample rates: soundcards, soundcard drivers, OS audio stack, or even the playback application can cause problems not related to the original audio, but to the different sampling rate being used.
- Volume mismatch: It is important that the samples at test are correctly leveled, so that the subject evaluates the audio, not the difference in audio amplitude. Differences of less than 1dB can be audible.
- Clipping/distortions caused by playback settings/equipment: It is not strange that audio clips digitally today. While the clip in itself could be heard and could determine a limitation or problem depending on the intended use, the problem could be gone just by setting the amplification properly.



When is an ABX necessary:

ABX test, as said above, help to determine that a difference exists. For Hydrogenaudio it is required when an *audible* difference is reported and:

- It is not commonly known that it exists
- It is against the general knowledge that says such difference should not be audible.
- It is not measurable by other means that could clearly determine otherwise (i.e. an audible difference in lossless codecs can be discarded if the codec works and so the problem, if any, exists somewhere else).



To finish this post, I'd like to add that ABX is not intended to avaluate if an item is better than another. In order to determine such thing, it is necessary that they are found to be different. Once that is verified, the difference can be avaluated between both items.

It's been said that ABX does not impose a limitation of how long/short a listening period runs for, neither puts it limits in how many times you switch back an forth to determine a difference/preference.
Also note that the existence of a difference is objective. The avaluation of the difference, while some rules apply, is more subjective.
That's why on the public listening tests, the statistic methods (like MUSHRA) that are applied on the final results of the group try to reduce the influence of subjective decisions.

This post has been edited by [JAZ]: Aug 11 2012, 16:32
Go to the top of the page
+Quote Post
audioclaudio
post Aug 11 2012, 19:15
Post #4





Group: Members
Posts: 33
Joined: 3-December 11
Member No.: 95560



QUOTE ([JAZ] @ Aug 11 2012, 16:20) *

neither puts it limits in how many times you switch back an forth to determine a difference/preference.

Had you actually even read the interview article, perhaps you would have known that switching back and forth is exactly the kind of thing that inevitably introduces bias except maybe if the test subjects are people who have the memory of a goldfish? rolleyes.gif
Go to the top of the page
+Quote Post
[JAZ]
post Aug 11 2012, 21:35
Post #5





Group: Members
Posts: 1797
Joined: 24-June 02
From: Catalunya(Spain)
Member No.: 2383



QUOTE (audioclaudio @ Aug 11 2012, 20:15) *
QUOTE ([JAZ] @ Aug 11 2012, 16:20) *

neither puts it limits in how many times you switch back an forth to determine a difference/preference.

Had you actually even read the interview article, perhaps you would have known that switching back and forth is exactly the kind of thing that inevitably introduces bias except maybe if the test subjects are people who have the memory of a goldfish? rolleyes.gif


Why are you always running away? I've said that it imposes no limits. It does not mean that you *have to*. And if that introduces bias, then, take a rest and come back.
There is nothing innerently wrong in asking the subject if it can correctly identify an item. The only limitations imposed by the ABX test is in avoiding conscious or unconscious preference for one of the items being tested.


Anyway, If that's the only thing of my reply that you find wrong, let's leave it that way and please, specify in which terms (examples or situations) a sighted test can be more reliable than a blind test, as you said in your last sentence, and which you completely avoided to answer at dhromed's request.

This post has been edited by [JAZ]: Aug 11 2012, 21:40
Go to the top of the page
+Quote Post
audioclaudio
post Aug 13 2012, 15:28
Post #6





Group: Members
Posts: 33
Joined: 3-December 11
Member No.: 95560



QUOTE ([JAZ] @ Aug 11 2012, 21:35) *
It does not mean that you *have to*.

I know that; in fact I never said the opposite.
QUOTE ([JAZ] @ Aug 11 2012, 21:35) *
And if that introduces bias, then, take a rest and come back.
Taking a rest doesn't magically erase my memory. Hence, taking a rest doesn't eliminate my cognitive bias. What happens if I listen to the same music on (1) system A, then (2) system B, then (3) system A again, is as follows. When I'm listening to system B, what I hear is something that's being cognitively biased as a result of what I can remember about how the music sounded on system A. First, let's assume system A is a less resolving system than B. While blinded, I can hear a detail with B that wan't noticeable at all with A. Still blinded, now let's assume I switch back to system A (according to step 3 in the experiment I described above). I can now still hear the detail. Why? Simple. It's because my memory is telling me that B sounded subjectively better, and that that's what the music should sound like. As a direct result from this, my brain now automatically fills in the missing detail where it previously couldn't. It previously could not, because the information required for that had previously not even been stored in my memory yet.
QUOTE ([JAZ] @ Aug 11 2012, 21:35) *

The only limitations imposed by the ABX test is in avoiding conscious or unconscious preference for one of the items being tested.

My above explanation, which is actually Bob Stuart's instead of mine (at least, if I understood the TAS article correctly...), shows that "avoiding conscious or unconscious preference" is not always technically possible if switching back and forth between audio samples is being allowed during these tests.
QUOTE ([JAZ] @ Aug 11 2012, 21:35) *

Anyway, If that's the only thing of my reply that you find wrong, let's leave it that way and please, specify in which terms (examples or situations) a sighted test can be more reliable than a blind test, as you said in your last sentence, and which you completely avoided to answer at dhromed's request.

"If you had the memory of a goldfish, maybe it would work." Seeing something can introduce bias because we can remember what we see, but that doesn't necessarily also mean hearing something can NEVER introduce bias. I can hear things while I'm blinded, things I can remember. I don't have to see what might be causing them to be able to remember that I heard them. So because I can remember them, I cannot always avoid being biased by them. If you don't believe that humans can be biased by things they can remember about a sound they heard, please watch the start of this vid.
http://www.youtube.com/watch?v=BYTlN6wjcvQ
Go to the top of the page
+Quote Post
dhromed
post Aug 13 2012, 16:46
Post #7





Group: Members
Posts: 1339
Joined: 16-February 08
From: NL
Member No.: 51347



QUOTE (audioclaudio @ Aug 13 2012, 16:28) *
I can hear things while I'm blinded, things I can remember. I don't have to see what might be causing them to be able to remember that I heard them. So because I can remember them, I cannot always avoid being biased by them.


That doesn't make a sighted test more reliable than a blind test. It's not an answer to my question, either.

QUOTE
can now still hear the detail. Why? Simple. It's because my memory is telling me that B sounded subjectively better, and that that's what the music should sound like. As a direct result from this, my brain now automatically fills in the missing detail where it previously couldn't. It previously could not, because the information required for that had previously not even been stored in my memory yet.


So your point is that due to this phantom detail, the ABX will show that A and B are equal, which would be wrong, since hypothetically, B is better. The blind test has failed.

Let's take that to its logical extreme: I play A, and there's nothing but silence. Then B, which contains music. Then I switch back to A, and because I've already heard B, my brain subconsciously superimposes the memory of the music onto the silence. I then falsely judge that A and B sound the same.

Obviously, you're not arguing that. So it becomes a question of magnitude. How tiny would a detail have to be before it can be superimposed on music without your conscious consent; before it messes up blind tests? If a detail is clearly audible, we can detect its absence when knowing it should be there; if it is not, then we hear nothing in either case.

It is a kind of defeatist stance: believing that objective methods are unreliable because the brain makes your perceive things that are not there. But the same goes for subjective methods! I've mentioned it before: it seems to come down to a crazy notion that things cannot be known and that knowledge can not be had.

This post has been edited by dhromed: Aug 13 2012, 16:50
Go to the top of the page
+Quote Post
audioclaudio
post Aug 13 2012, 18:39
Post #8





Group: Members
Posts: 33
Joined: 3-December 11
Member No.: 95560



QUOTE (dhromed @ Aug 13 2012, 16:46) *
QUOTE (audioclaudio @ Aug 13 2012, 16:28) *
I can hear things while I'm blinded, things I can remember. I don't have to see what might be causing them to be able to remember that I heard them. So because I can remember them, I cannot always avoid being biased by them.


That doesn't make a sighted test more reliable than a blind test. It's not an answer to my question, either.

I agree. It only shows that both sighted and double blind tests can be, but not necessarily always are, unreliable. If there can be no reliable way to show that a test is indeed reliable, logically we must accept the possibility that it's not. The test being blinded or sighted does not change this fact. In answer to your question, here is the link to another article about this subject. As you can see, it has been debated over to great length.
http://www.avguide.com/forums/blind-listen...lawed-editorial
QUOTE (dhromed @ Aug 13 2012, 16:46) *
So your point is that due to this phantom detail, the ABX will show that A and B are equal, which would be wrong, since hypothetically, B is better. The blind test has failed.

Let's take that to its logical extreme: I play A, and there's nothing but silence. Then B, which contains music. Then I switch back to A, and because I've already heard B, my brain subconsciously superimposes the memory of the music onto the silence. I then falsely judge that A and B sound the same.

Obviously, you're not arguing that. So it becomes a question of magnitude. How tiny would a detail have to be before it can be superimposed on music without your conscious consent; before it messes up blind tests? If a detail is clearly audible, we can detect its absence when knowing it should be there; if it is not, then we hear nothing in either case.

It is a kind of defeatist stance: believing that objective methods are unreliable because the brain makes your perceive things that are not there. But the same goes for subjective methods! I've mentioned it before: it seems to come down to a crazy notion that things cannot be known and that knowledge can not be had.

Well, yes. It IMO is not a crazy notion at all, though. Take neuroscience for example, it's not an exact science. In fact, if all knowledge could always be had then perhaps we wouldn't be needing science in the first place. After all, the goal of science is not only to obtain knowledge, but to at least TRY and obtain knowledge, so here's yet another lovely anecdote. One cannot prove the existence of the Higgs boson until one actually DOES find it. But then, how are people supposed to find the little bugger if they all just refuse to even keep looking?
Go to the top of the page
+Quote Post
Arnold B. Kruege...
post Aug 13 2012, 19:35
Post #9





Group: Members
Posts: 4457
Joined: 29-October 08
From: USA, 48236
Member No.: 61311



QUOTE (audioclaudio @ Aug 13 2012, 13:39) *
In answer to your question, here is the link to another article about this subject. As you can see, it has been debated over to great length.
http://www.avguide.com/forums/blind-listen...lawed-editorial


Interesting that you cite that article. What you find there is not an open honest debate of the topic, but rather a sham discussion where many critical comments were censured out of existence.
Go to the top of the page
+Quote Post

Posts in this topic
2 Pages V   1 2 >


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 20th December 2014 - 12:49