IPB

Welcome Guest ( Log In | Register )

2 Pages V   1 2 >  
Reply to this topicStart new topic
ABX Debate
ching-3
post Aug 21 2005, 21:15
Post #1





Group: Members
Posts: 53
Joined: 6-August 05
From: Northern Ireland, UK
Member No.: 23741



Recently I posted on this thread:

http://www.hydrogenaudio.org/forums/index....showtopic=36432

I quickly realised I'd like to begin a discussion / debate on ABX.
Now, before I get reported for trolling, I would like to say that I use ABX. I'm not against using it.

The debate is simple. Is it okay to rely _soley_ on ABX testing? More technically: If two samples are ABXed and the results indicates that the listener cannot tell the difference, can this be _defined_ as transparent?

Someone once pointed out that choosing in ABX involves concious thought. Listening however also involves the subconsiousness.

Let the mayhem begin tongue.gif


--------------------
You messed up, now I gotta mess you up. It's the law!
Go to the top of the page
+Quote Post
bug80
post Aug 21 2005, 21:26
Post #2





Group: Members
Posts: 398
Joined: 23-January 05
From: The Netherlands
Member No.: 19254



QUOTE (ching-3 @ Aug 21 2005, 10:15 PM)
The debate is simple. Is it okay to rely _soley_ on ABX testing? More technically: If two samples are ABXed and the results indicates that the listener cannot tell the difference, can this be _defined_ as transparent?


As far as I know, with an ABX test, you can only prove that there is a difference, not that there is no difference.

If that was possible, I could simply prove that an encoder is transparent by not listening carefully during the test (failing the ABX test on purpose).

So the hypothesis is: "There is a difference between A and B" and it is up to the tester to prove it. Two samples are the same unless proven otherwise smile.gif
Go to the top of the page
+Quote Post
skelly831
post Aug 21 2005, 21:26
Post #3





Group: Members (Donating)
Posts: 782
Joined: 11-April 05
From: México
Member No.: 21361



QUOTE (ching-3 @ Aug 21 2005, 01:15 PM)
The debate is simple. Is it okay to rely _soley_ on ABX testing? More technically: If two samples are ABXed and the results indicates that the listener cannot tell the difference, can this be _defined_ as transparent?


It can be defined as "transparent" for that particular person. Thats why people shouldn't always follow recommendations blindly, they should make the effort to find their particular format settings for what they descover is transparent to them, and for this you need ABX testing.

This post has been edited by skelly831: Aug 21 2005, 21:27


--------------------
we was young an' full of beans
Go to the top of the page
+Quote Post
Lyx
post Aug 21 2005, 21:27
Post #4





Group: Members
Posts: 3353
Joined: 6-July 03
From: Sachsen (DE)
Member No.: 7609



QUOTE (ching-3 @ Aug 21 2005, 10:15 PM)
If two samples are ABXed and the results indicates that the listener cannot tell the difference, can this be _defined_ as transparent?
*

No, because its impossible to "prove" that something does *not* exist. It is only possible to prove that something *does* exist.

So, strictly spoken, you can never be 100% sure that something *is* transparent. But you also can never be sure that in 3mins weird aliens will not kidnapp you ;-) It all comes down to probabilities - if a significant amount of skilled testers were unable to prove that there is a difference, then it is probable that the majority of people also cannot detect a difference - this is what usually is called "transparent".

Thus, there is no such thing as absolute certainity. You trust/make decisions based on probabilities - not on "real" certainities. Thats not something specific to audio - you handle things like this countless times per day. For example, if you would only cross the street if it is 100% safe, then you could never cross it - because there would always be the possibility that you erred. So, based on probabilities you "hypothetically asume" that something is certain - the problem is just that people tend to forget that this "certainity" is only hypothetical.

Personal opinion following:
Concerning objective vs. subjective methods, it depends on the case. There is a reason why humans have heart as well as mind - because both have their individual strengths and weaknesses. The mind tends to be more "error-safe" but is also more restricted. Heart and intuition can be useful in unknown situations but are more prone to error. If heart, mind or a combo is most effective depends on the case.... with the topics discussed on ha.org, a scientific approach(mind) is more efficient imho, and thats why i do agree with TOS#8. Subjective means *in this case* would do more harm than good.

This post has been edited by Lyx: Aug 21 2005, 21:44


--------------------
I am arrogant and I can afford it because I deliver.
Go to the top of the page
+Quote Post
mdefranc
post Aug 21 2005, 23:52
Post #5





Group: Members (Donating)
Posts: 49
Joined: 15-October 01
From: Midwest
Member No.: 295



"Listening however also involves the subconsiousness." - ching3

How are you conscious of this subconsiousness?
Go to the top of the page
+Quote Post
krazy
post Aug 22 2005, 10:29
Post #6





Group: Members
Posts: 493
Joined: 3-June 03
Member No.: 6981



I think 'transparent', when referring to lossy codecs, is a great simplification and a confusing term to use. IMHO a more correct term (variations of which is repeated throughout this thread/ha.org) would be 'effectively transparent for this person and sample'.

Popper said:
QUOTE
No matter how many instances of white swans we may have observed, this does not justify the conclusion that all swans are white.


Thus in audio encoding the only thing which can be defined as transparent is lossless.
Go to the top of the page
+Quote Post
bryant
post Aug 22 2005, 17:56
Post #7


WavPack Developer


Group: Developer (Donating)
Posts: 1290
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



QUOTE (mdefranc @ Aug 21 2005, 02:52 PM)
"Listening however also involves the subconsiousness." - ching3

How are you conscious of this subconsiousness?
*

Here are a couple of interesting research papers in the area of perception.

The first is an overview of research regarding conscious vs. unconscious perception, where most of the work has been done with the visual system:

http://watarts.uwaterloo.ca/~pmerikle/pape...tudies.1998.pdf

This one is focused on the auditory system, specifically how we might perceive and react to rhythm (consciously and unconsciously):

http://www.apa.org/journals/features/xhp273600.pdf
Go to the top of the page
+Quote Post
henkersmahlzeit
post Aug 22 2005, 19:52
Post #8





Group: Members
Posts: 110
Joined: 31-December 03
Member No.: 10840



QUOTE (bryant @ Aug 22 2005, 08:56 AM)
Here are a couple of interesting research papers in the area of perception.

The first is an overview of research regarding conscious vs. unconscious perception, where most of the work has been done with the visual system:

http://watarts.uwaterloo.ca/~pmerikle/pape...tudies.1998.pdf

This one is focused on the auditory system, specifically how we might perceive and react to rhythm (consciously and unconsciously):

http://www.apa.org/journals/features/xhp273600.pdf
*

Wow! Thanx for the references!
Go to the top of the page
+Quote Post
Nero
post Aug 23 2005, 06:53
Post #9





Group: Members
Posts: 47
Joined: 16-March 05
Member No.: 20681



QUOTE (ching-3 @ Aug 21 2005, 04:15 PM)
The debate is simple. Is it okay to rely _soley_ on ABX testing? More technically: If two samples are ABXed and the results indicates that the listener cannot tell the difference, can this be _defined_ as transparent?
*

It's generally a better idea in a task of analysis to rely on as many reliable sources as you can. But ABX is among the best sources because it uses double-blind testing to generate empirical results. And yes, if the listener cannot differentiate between the compressed test sample and the uncompressed reference in enough test cycles to be meaningful, the compressed encoding is shown to be perceptually transparent to that listener.

QUOTE (mdefranc @ Aug 21 2005, 06:52 PM)
"Listening however also involves the subconsiousness." - ching3

How are you conscious of this subconsiousness?
*

Because you can hear when you're not conscious. This isn't clear proof, only an indication in that direction.

QUOTE (krazy @ Aug 22 2005, 05:29 AM)
I think 'transparent', when referring to lossy codecs, is a great simplification and a confusing term to use. IMHO a more correct term (variations of which is repeated throughout this thread/ha.org) would be 'effectively transparent for this person and sample'.
*

What I generally say is "perceptually transparent for the listener", and just saying "transparent" in this context is generally understood to mean the same thing.

QUOTE (krazy @ Aug 22 2005, 05:29 AM)
Popper said:
QUOTE
No matter how many instances of white swans we may have observed, this does not justify the conclusion that all swans are white.


Thus in audio encoding the only thing which can be defined as transparent is lossless.
*


That would only be the case if the context were either infinite or so vast as to not be reasonably measured. Any scientific analysis must have a defined scope to be meaningful. One person with a measureable hearing ability using ABX to test a few seconds of music for perceptual transparency can certainly identify what is or is not transparent to their ears within the scope of (1) the sample being tested, (2) the encoding format and setting used, (3) the individual's health and hearing ability, (4) the rigorousness of the test (i.e., the listener follows the rules of the test and applies all of their abilities) and (5) the number of test cycles executed.


--------------------
Sometimes you have to jump off the cliff and build your wings on the way down.
Go to the top of the page
+Quote Post
woody_woodward
post Aug 23 2005, 17:38
Post #10





Group: Members
Posts: 348
Joined: 21-August 02
Member No.: 3138



QUOTE (Nero @ Aug 22 2005, 09:53 PM)
That would only be the case if the context were either infinite or so vast as to not be reasonably measured.  Any scientific analysis must have a defined scope to be meaningful.  One person with a measureable hearing ability using ABX to test a few seconds of music for perceptual transparency can certainly identify what is or is not transparent to their ears within the scope of (1) the sample being tested, (2) the encoding format and setting used, (3) the individual's health and hearing ability, (4) the rigorousness of the test (i.e., the listener follows the rules of the test and applies all of their abilities) and (5) the number of test cycles executed.
*

... and (6) the playback environment and equipment being used.

I guess the bottom line here is that while ABX is an excellent tool, but not perfect. Also, if the masses hear things one way and I hear something else, then we do have a clear winner: C'est moi !
Go to the top of the page
+Quote Post
Nero
post Aug 23 2005, 18:38
Post #11





Group: Members
Posts: 47
Joined: 16-March 05
Member No.: 20681



True that you need a quiet and controlled environment, but I was thinking this is provided in the "rules" of using an ABX comparator. I could certainly be wrong on that, though.

But the equipment used really doesn't matter as much as people may think. In my experience the vast majority of artifacts that can be heard on a $10,000 system can just as well be heard on an off-brand sound card and $8 earbuds.


--------------------
Sometimes you have to jump off the cliff and build your wings on the way down.
Go to the top of the page
+Quote Post
ching-3
post Aug 23 2005, 21:16
Post #12





Group: Members
Posts: 53
Joined: 6-August 05
From: Northern Ireland, UK
Member No.: 23741



I see, well I'm guessing now for ABX to be trully perfect, one would have to run through an infinite number of trials! unsure.gif

But of course it seems ABX is very reliable, despite minor uncertainties.

Isn't there some kind of method in scientific research that goes along the lines of 'prove this is incorrect and thus the opposite must be correct'...? Something to do with an anti hypothesis, I learnt this in high school, but it's all very fuzzy tongue.gif Perhaps there is a way to be (nearly) certain that an artifact or difference is not noticible, ever, by the listener...

Edit: Punctuation

This post has been edited by ching-3: Aug 23 2005, 21:20


--------------------
You messed up, now I gotta mess you up. It's the law!
Go to the top of the page
+Quote Post
stephanV
post Aug 23 2005, 21:52
Post #13





Group: Members
Posts: 394
Joined: 6-May 04
Member No.: 13932



QUOTE
I see, well I'm guessing now for ABX to be trully perfect, one would have to run through an infinite number of trials!

The outcome ABXing is a probability, not a "yes" or "no". There is no perfectness in that.

QUOTE
Isn't there some kind of method in scientific research that goes along the lines of 'prove this is incorrect and thus the opposite must be correct'...?

In a way this is already what ABX is about. You are trying to statisticaly prove that you CAN hear a difference. If you fail in statistically proving this, the opposite must be true: you CANNOT hear a difference.

QUOTE
Perhaps there is a way to be (nearly) certain that an artifact or difference is not noticible, ever, by the listener...

ABX can do that for one listener, but it wouldn't be fair to extrapolate this answer to everybody else. Even when a group of listeners has the same ABX result, you cannot be absolutely sure that this will count for everybody else.

A certain uncertainty will always remain in this, you will just have to decide for yourself with how much incertainty you want to live.

As for this comment:
QUOTE
Someone once pointed out that choosing in ABX involves concious thought. Listening however also involves the subconsiousness.

If your subconsious would play a role in normal life, why not during an ABX test?


--------------------
"We cannot win against obsession. They care, we don't. They win."
Go to the top of the page
+Quote Post
Lyx
post Aug 23 2005, 22:21
Post #14





Group: Members
Posts: 3353
Joined: 6-July 03
From: Sachsen (DE)
Member No.: 7609



QUOTE
Perhaps there is a way to be (nearly) certain that an artifact or difference is not noticible, ever, by the listener...

No - this is generally not possible, because it is impossible to prove that something doesn't exist. Absolute certainity simply is not achievable. All you can do in your daily life as well as science is to rate probabilities - define a degree of unprobability at which "it is quite safe to asume that it doesn't exist".

In the end, we are all just gamblers who try to get the most safe deal. But some uncertainity is part of the game ;)

QUOTE (stephanV @ Aug 23 2005, 10:52 PM)
A certain uncertainty will always remain in this, you will just have to decide for yourself with how much incertainty you want to live.

Exactly.


--------------------
I am arrogant and I can afford it because I deliver.
Go to the top of the page
+Quote Post
Lyx
post Aug 24 2005, 01:12
Post #15





Group: Members
Posts: 3353
Joined: 6-July 03
From: Sachsen (DE)
Member No.: 7609



QUOTE (stephanV @ Aug 23 2005, 10:52 PM)
QUOTE
Someone once pointed out that choosing in ABX involves concious thought. Listening however also involves the subconsiousness.

If your subconsious would play a role in normal life, why not during an ABX test?
*


I think he just worded a bit wrong what he was trying to say. What he probably meant is that a "quick ABX-test" leaves out mid/longterm-feedback.

Nothing stops one from doing relaxed longterm ABX-tests which could als include subconscious "feedback" (some would call that intuition or just "feeling"). But in practice, such tests are rare because the effort and time needed is multiple times higher than with a "quick" ABX.

So, its not a limitation of the ABX-methodology. Its a limitation of how much effort people are willing to put into such a test.

- Lyx

This post has been edited by Lyx: Aug 24 2005, 01:14


--------------------
I am arrogant and I can afford it because I deliver.
Go to the top of the page
+Quote Post
IgorC
post Jan 12 2006, 19:44
Post #16





Group: Members
Posts: 1532
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Is there any sofware to abx for videocodecs?
I know it's audiorelated so I created a new topic here. I don't want to crosspost but guys who are interesting about audio compression don't open this subforum frequently. smile.gif

http://www.hydrogenaudio.org/forums/index....showtopic=40537
Go to the top of the page
+Quote Post
MLP
post Jan 12 2006, 20:41
Post #17





Group: Members
Posts: 16
Joined: 15-December 05
Member No.: 26423



Excuse me guys, I'm a newbie but there is something I can't understand.

- On one hand you can measure what you can hear. I mean, there are "machines" that can measure the frequencies that an specific human can hear. That´s objective.

- On the other hand you have a song encoded at x kbps that at the end of the day (correct me if I'm wrong) will deliver different kind of frequencies. This element is also objective.

Honestly, I know nothing about ABX test but by definition, some frequencies of the song will fall within the range you can hear, and some won't.

What I'm trying to say, is that despite the fact that I can say that I cannot hear a difference in an ABX test, objectively I must hear it if my ears are able to process the frequencies. Am I so wrong?
Go to the top of the page
+Quote Post
saratoga
post Jan 12 2006, 20:55
Post #18





Group: Members
Posts: 4845
Joined: 2-September 02
Member No.: 3264



QUOTE (MLP @ Jan 12 2006, 12:41 PM)
Excuse me guys, I'm a newbie but there is something I can't understand.

- On one hand you can measure what you can hear. I mean, there are "machines" that can measure the frequencies that an specific human can hear. That´s objective.

- On the other hand you have a song encoded at x kbps that at the end of the day (correct me if I'm wrong) will deliver different kind of frequencies. This element is also objective.

Honestly, I know nothing about ABX test but by definition, some frequencies of the song will fall within the range you can hear, and some won't.

What I'm trying to say, is that despite the fact that I can say that I cannot hear a difference in an ABX test, objectively I must hear it if my ears are able to process the frequencies. Am I so wrong?
*


Yes you are wrong. Your ears/brain are only processing some information based on a complicated set of rules and their own physical limitations. Furthermore, while you may be able to hear all frequencies within a band, you may not hear certain frequencies under certain conditions (for instance, very quiet sounds may not be noticed in a noisy room).

If you can hear the difference, you should eventually be able to ABX the difference given enough time and motivation.
Go to the top of the page
+Quote Post
Woodinville
post Jan 15 2006, 12:27
Post #19





Group: Members
Posts: 1402
Joined: 9-January 05
From: JJ's office.
Member No.: 18957



QUOTE (ching-3 @ Aug 21 2005, 12:15 PM)
The debate is simple. Is it okay to rely _soley_ on ABX testing? More technically: If two samples are ABXed and the results indicates that the listener cannot tell the difference, can this be _defined_ as transparent?
*


All you can do is say that "in this test, the listener(s) did not make any distinction". No more, no less.


--------------------
-----
J. D. (jj) Johnston
Go to the top of the page
+Quote Post
Digga
post Jan 15 2006, 12:43
Post #20





Group: Members
Posts: 1047
Joined: 28-June 03
From: on the dock of the bay
Member No.: 7423



QUOTE (Woodinville @ Jan 15 2006, 12:27 PM)
All you can do is say that "in this test, the listener(s) did not make any distinction". No more, no less.
no. if n (listeners) is high enough and tests are held under reproducible and standardized circumstances minimizing other variables that might interfere with the results then you can say with a certain amount of confidence (e.g. 95%) that results are valid to the whole population.


--------------------
Nothing but a Heartache - Since I found my Baby ;)
Go to the top of the page
+Quote Post
HotshotGG
post Jan 15 2006, 12:54
Post #21





Group: Members
Posts: 1593
Joined: 24-March 02
From: Revere, MA
Member No.: 1607



Since when has Hydrogenaudio become the birthplace of "magical thinking"? It's kind of like attempting to argue the case of Free Will Vs. Determinism. I personally just like to settle for Compatiblism and call it a day. biggrin.gif


--------------------
College student/IT Assistant
Go to the top of the page
+Quote Post
cabbagerat
post Jan 15 2006, 14:08
Post #22





Group: Members
Posts: 1018
Joined: 27-September 03
From: Cape Town
Member No.: 9042



QUOTE (Woodinville @ Jan 15 2006, 03:27 AM)
All you can do is say that "in this test, the listener(s) did not make any distinction". No more, no less.
*
I agree.
QUOTE (Digga @ Jan 15 2006, 03:43 AM)
no. if n (listeners) is high enough and tests are held under reproducible and standardized circumstances minimizing other variables that might interfere with the results then you can say with a certain amount of confidence (e.g. 95%) that results are valid to the whole population.
*
I disagree. Because of the structure of ABX testing, the negative case (no difference can be heard) is can be trivially affected by expectation bias. For the positive case, however, it reduces the effect of expectation bias to an insignificant level. For example, If I am comparing 32kbps MP3 versus 44.1/16 wav and I truly expect there not to be a difference, I can bias the result of the test in such a way as a negative result is produced.

Surely, based on this, you can't prove a negative hypothesis with the ABX method. If you think I am wrong, please share your reasons.


--------------------
Simulate your radar: http://www.brooker.co.za/fers/
Go to the top of the page
+Quote Post
Woodinville
post Jan 16 2006, 03:15
Post #23





Group: Members
Posts: 1402
Joined: 9-January 05
From: JJ's office.
Member No.: 18957



QUOTE (Digga @ Jan 15 2006, 03:43 AM)
QUOTE (Woodinville @ Jan 15 2006, 12:27 PM)
All you can do is say that "in this test, the listener(s) did not make any distinction". No more, no less.
no. if n (listeners) is high enough and tests are held under reproducible and standardized circumstances minimizing other variables that might interfere with the results then you can say with a certain amount of confidence (e.g. 95%) that results are valid to the whole population.
*




You can always sample population, but you can never make more than a probabilistic statement in either direction, please.


--------------------
-----
J. D. (jj) Johnston
Go to the top of the page
+Quote Post
Woodinville
post Jan 16 2006, 03:17
Post #24





Group: Members
Posts: 1402
Joined: 9-January 05
From: JJ's office.
Member No.: 18957



QUOTE (cabbagerat @ Jan 15 2006, 05:08 AM)
Surely, based on this, you can't prove a negative hypothesis with the ABX method. If you think I am wrong, please share your reasons.
*



You can use controls to test the sensitivity of any given test. You can make a probabilistic statement based on any given test. There is no "proof" in any case, either way, only stronger and stronger (as results and 'n' go up) probabilistic proof.

I personally take "proof" to mean what one "proves" in a mathematical proof, and there is no absolute certainty in subjective testing, only probabilistic certainty.


--------------------
-----
J. D. (jj) Johnston
Go to the top of the page
+Quote Post
ShowsOn
post Jan 16 2006, 03:56
Post #25





Group: Members (Donating)
Posts: 369
Joined: 28-June 02
From: South Australia, AUSTRALIA
Member No.: 2421



*

QUOTE
For example, If I am comparing 32kbps MP3 versus 44.1/16 wav and I truly expect there not to be a difference, I can bias the result of the test in such a way as a negative result is produced.

Does that count as properly performing an ABX test - deliberately saying the files are indistiguishable when you can actually hear a difference (selecting B when you know the correct answer is A)?

To me that is more akin to making a concerted attempt to bias/sabotage the test. Any experiment will become useless if testers makes a decision to ignore, or say the opposite, to what their senses are telling them.

If a sufficient group of testers perform the same test, it is unlikely that all of them will share the same bias, and thus it is likely that more accurate results will out-weigh what would be later considered an anomaly.

If a hundred people are asked what colour the sky is on a clear day, and I say red, knowing I actually see blue, yet the other 99 people say that it is blue, then we can be a lot more confident the sky is blue, rather than red. The 'bad' data gets filtered out in the end.


--------------------
www.petitiononline.com/RHCPWBCD/petition.html
Go to the top of the page
+Quote Post

2 Pages V   1 2 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 24th July 2014 - 00:45