Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: ABX Debate (Read 17647 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

ABX Debate

Recently I posted on this thread:

http://www.hydrogenaudio.org/forums/index....showtopic=36432

I quickly realised I'd like to begin a discussion / debate on ABX.
Now, before I get reported for trolling, I would like to say that I use ABX. I'm not against using it.

The debate is simple. Is it okay to rely _soley_ on ABX testing? More technically: If two samples are ABXed and the results indicates that the listener cannot tell the difference, can this be _defined_ as transparent?

Someone once pointed out that choosing in ABX involves concious thought. Listening however also involves the subconsiousness.

Let the mayhem begin 
You messed up, now I gotta mess you up. It's the law!

ABX Debate

Reply #1 –
Quote
The debate is simple. Is it okay to rely _soley_ on ABX testing? More technically: If two samples are ABXed and the results indicates that the listener cannot tell the difference, can this be _defined_ as transparent?


As far as I know, with an ABX test, you can only prove that there is a difference, not that there is no difference.

If that was possible, I could simply prove that an encoder is transparent by not listening carefully during the test (failing the ABX test on purpose).

So the hypothesis is: "There is a difference between A and B" and it is up to the tester to prove it. Two samples are the same unless proven otherwise

ABX Debate

Reply #2 –
Quote
The debate is simple. Is it okay to rely _soley_ on ABX testing? More technically: If two samples are ABXed and the results indicates that the listener cannot tell the difference, can this be _defined_ as transparent?


It can be defined as "transparent" for that particular person. Thats why people shouldn't always follow recommendations blindly, they should make the effort to find their particular format settings for what they descover is transparent to them, and for this you need ABX testing.
we was young an' full of beans

ABX Debate

Reply #3 –
Quote
If two samples are ABXed and the results indicates that the listener cannot tell the difference, can this be _defined_ as transparent?
[a href="index.php?act=findpost&pid=321579"][{POST_SNAPBACK}][/a]

No, because its impossible to "prove" that something does *not* exist. It is only possible to prove that something *does* exist.

So, strictly spoken, you can never be 100% sure that something *is* transparent. But you also can never be sure that in 3mins weird aliens will not kidnapp you ;-) It all comes down to probabilities - if a significant amount of skilled testers were unable to prove that there is a difference, then it is probable that the majority of people also cannot detect a difference - this is what usually is called "transparent".

Thus, there is no such thing as absolute certainity. You trust/make decisions based on probabilities - not on "real" certainities. Thats not something specific to audio - you handle things like this countless times per day. For example, if you would only cross the street if it is 100% safe, then you could never cross it - because there would always be the possibility that you erred. So, based on probabilities you "hypothetically asume" that something is certain - the problem is just that people tend to forget that this "certainity" is only hypothetical.

Personal opinion following:
Concerning objective vs. subjective methods, it depends on the case. There is a reason why humans have heart as well as mind - because both have their individual strengths and weaknesses. The mind tends to be more "error-safe" but is also more restricted. Heart and intuition can be useful in unknown situations but are more prone to error. If heart, mind or a combo is most effective depends on the case.... with the topics discussed on ha.org, a scientific approach(mind) is more efficient imho, and thats why i do agree with TOS#8. Subjective means *in this case* would do more harm than good.
I am arrogant and I can afford it because I deliver.

ABX Debate

Reply #4 –
"Listening however also involves the subconsiousness." - ching3

How are you conscious of this subconsiousness?

ABX Debate

Reply #5 –
I think 'transparent', when referring to lossy codecs, is a great simplification and a confusing term to use. IMHO a more correct term (variations of which is repeated throughout this thread/ha.org) would be 'effectively transparent for this person and sample'.

Popper said:
Quote
No matter how many instances of white swans we may have observed, this does not justify the conclusion that all swans are white.


Thus in audio encoding the only thing which can be defined as transparent is lossless.

ABX Debate

Reply #6 –
Quote
"Listening however also involves the subconsiousness." - ching3

How are you conscious of this subconsiousness?
[{POST_SNAPBACK}][/a]

Here are a couple of interesting research papers in the area of perception.

The first is an overview of research regarding conscious vs. unconscious perception, where most of the work has been done with the visual system:

[a href="http://watarts.uwaterloo.ca/~pmerikle/papers/Merikle.JConsStudies.1998.pdf]http://watarts.uwaterloo.ca/~pmerikle/pape...tudies.1998.pdf[/url]

This one is focused on the auditory system, specifically how we might perceive and react to rhythm (consciously and unconsciously):

http://www.apa.org/journals/features/xhp273600.pdf

ABX Debate

Reply #7 –
Quote
Here are a couple of interesting research papers in the area of perception.

The first is an overview of research regarding conscious vs. unconscious perception, where most of the work has been done with the visual system:

http://watarts.uwaterloo.ca/~pmerikle/pape...tudies.1998.pdf

This one is focused on the auditory system, specifically how we might perceive and react to rhythm (consciously and unconsciously):

http://www.apa.org/journals/features/xhp273600.pdf
[a href="index.php?act=findpost&pid=321782"][{POST_SNAPBACK}][/a]

Wow! Thanx for the references!

ABX Debate

Reply #8 –
Quote
The debate is simple. Is it okay to rely _soley_ on ABX testing? More technically: If two samples are ABXed and the results indicates that the listener cannot tell the difference, can this be _defined_ as transparent?
[{POST_SNAPBACK}][/a]

It's generally a better idea in a task of analysis to rely on as many reliable sources as you can.  But ABX is among the best sources because it uses double-blind testing to generate empirical results.  And yes, if the listener cannot differentiate between the compressed test sample and the uncompressed reference in enough test cycles to be meaningful, the compressed encoding is shown to be perceptually transparent to that listener.

Quote
"Listening however also involves the subconsiousness." - ching3

How are you conscious of this subconsiousness?
[a href="index.php?act=findpost&pid=321621"][{POST_SNAPBACK}][/a]

Because you can hear when you're not conscious.  This isn't clear proof, only an indication in that direction.

Quote
I think 'transparent', when referring to lossy codecs, is a great simplification and a confusing term to use. IMHO a more correct term (variations of which is repeated throughout this thread/ha.org) would be 'effectively transparent for this person and sample'.
[a href="index.php?act=findpost&pid=321695"][{POST_SNAPBACK}][/a]

What I generally say is "perceptually transparent for the listener", and just saying "transparent" in this context is generally understood to mean the same thing.

Quote
[a href="http://en.wikipedia.org/wiki/Karl_Popper]Popper[/url] said:
Quote
No matter how many instances of white swans we may have observed, this does not justify the conclusion that all swans are white.


Thus in audio encoding the only thing which can be defined as transparent is lossless.
[a href="index.php?act=findpost&pid=321695"][{POST_SNAPBACK}][/a]

That would only be the case if the context were either infinite or so vast as to not be reasonably measured.  Any scientific analysis must have a defined scope to be meaningful.  One person with a measureable hearing ability using ABX to test a few seconds of music for perceptual transparency can certainly identify what is or is not transparent to their ears within the scope of (1) the sample being tested, (2) the encoding format and setting used, (3) the individual's health and hearing ability, (4) the rigorousness of the test (i.e., the listener follows the rules of the test and applies all of their abilities) and (5) the number of test cycles executed.
Sometimes you have to jump off the cliff and build your wings on the way down.

ABX Debate

Reply #9 –
Quote
That would only be the case if the context were either infinite or so vast as to not be reasonably measured.  Any scientific analysis must have a defined scope to be meaningful.  One person with a measureable hearing ability using ABX to test a few seconds of music for perceptual transparency can certainly identify what is or is not transparent to their ears within the scope of (1) the sample being tested, (2) the encoding format and setting used, (3) the individual's health and hearing ability, (4) the rigorousness of the test (i.e., the listener follows the rules of the test and applies all of their abilities) and (5) the number of test cycles executed.
[a href="index.php?act=findpost&pid=321956"][{POST_SNAPBACK}][/a]

... and (6) the playback environment and equipment being used.

I guess the bottom line here is that while ABX is an excellent tool, but not perfect.  Also, if the masses hear things one way and I hear something else, then we do have a clear winner:  C'est moi !

ABX Debate

Reply #10 –
True that you need a quiet and controlled environment, but I was thinking this is provided in the "rules" of using an ABX comparator.  I could certainly be wrong on that, though.

But the equipment used really doesn't matter as much as people may think.  In my experience the vast majority of artifacts that can be heard on a $10,000 system can just as well be heard on an off-brand sound card and $8 earbuds.
Sometimes you have to jump off the cliff and build your wings on the way down.

ABX Debate

Reply #11 –
I see, well I'm guessing now for ABX to be trully perfect, one would have to run through an infinite number of trials! 

But of course it seems ABX is very reliable, despite minor uncertainties.

Isn't there some kind of method in scientific research that goes along the lines of 'prove this is incorrect and thus the opposite must be correct'...? Something to do with an anti hypothesis, I learnt this in high school, but it's all very fuzzy  Perhaps there is a way to be (nearly) certain that an artifact or difference is not noticible, ever, by the listener...

Edit: Punctuation
You messed up, now I gotta mess you up. It's the law!

ABX Debate

Reply #12 –
Quote
I see, well I'm guessing now for ABX to be trully perfect, one would have to run through an infinite number of trials!

The outcome ABXing is a probability, not a "yes" or "no". There is no perfectness in that.

Quote
Isn't there some kind of method in scientific research that goes along the lines of 'prove this is incorrect and thus the opposite must be correct'...?

In a way this is already what ABX is about. You are trying to statisticaly prove that you CAN hear a difference. If you fail in statistically proving this, the opposite must be true: you CANNOT hear a difference.

Quote
Perhaps there is a way to be (nearly) certain that an artifact or difference is not noticible, ever, by the listener...

ABX can do that for one listener, but it wouldn't be fair to extrapolate this answer to everybody else. Even when a group of listeners has the same ABX result, you cannot be absolutely sure that this will count for everybody else.

A certain uncertainty will always remain in this, you will just have to decide for yourself with how much incertainty you want to live.

As for this comment:
Quote
Someone once pointed out that choosing in ABX involves concious thought. Listening however also involves the subconsiousness.

If your subconsious would play a role in normal life, why not during an ABX test?
"We cannot win against obsession. They care, we don't. They win."

ABX Debate

Reply #13 –
Quote
Perhaps there is a way to be (nearly) certain that an artifact or difference is not noticible, ever, by the listener...

No - this is generally not possible, because it is impossible to prove that something doesn't exist. Absolute certainity simply is not achievable. All you can do in your daily life as well as science is to rate probabilities - define a degree of unprobability at which "it is quite safe to asume that it doesn't exist".

In the end, we are all just gamblers who try to get the most safe deal. But some uncertainity is part of the game ;)

Quote
A certain uncertainty will always remain in this, you will just have to decide for yourself with how much incertainty you want to live.

Exactly.
I am arrogant and I can afford it because I deliver.

ABX Debate

Reply #14 –
Quote
Quote
Someone once pointed out that choosing in ABX involves concious thought. Listening however also involves the subconsiousness.

If your subconsious would play a role in normal life, why not during an ABX test?
[a href="index.php?act=findpost&pid=322156"][{POST_SNAPBACK}][/a]

I think he just worded a bit wrong what he was trying to say. What he probably meant is that a "quick ABX-test" leaves out mid/longterm-feedback.

Nothing stops one from doing relaxed longterm ABX-tests which could als include subconscious "feedback" (some would call that intuition or just "feeling"). But in practice, such tests are rare because the effort and time needed is multiple times higher than with a "quick" ABX.

So, its not a limitation of the ABX-methodology. Its a limitation of how much effort people are willing to put into such a test.

- Lyx
I am arrogant and I can afford it because I deliver.

ABX Debate

Reply #15 –
Is there any sofware to abx for videocodecs?
I know it's audiorelated so I created a new topic here. I don't want to crosspost but guys who are interesting about audio compression don't open this subforum frequently. 

http://www.hydrogenaudio.org/forums/index....showtopic=40537

ABX Debate

Reply #16 –
Excuse me guys, I'm a newbie but there is something I can't understand.

- On one hand you can measure what you can hear. I mean, there are "machines" that can measure the frequencies that an specific human can hear. That´s objective.

- On the other hand you have a song encoded at x kbps that at the end of the day (correct me if I'm wrong) will deliver different kind of frequencies. This element is also objective.

Honestly, I know nothing about ABX test but by definition, some frequencies of the song will fall within the range you can hear, and some won't.

What I'm trying to say, is that despite the fact that I can say that I cannot hear a difference in an ABX test, objectively I must hear it if my ears are able to process the frequencies. Am I so wrong?

ABX Debate

Reply #17 –
Quote
Excuse me guys, I'm a newbie but there is something I can't understand.

- On one hand you can measure what you can hear. I mean, there are "machines" that can measure the frequencies that an specific human can hear. That´s objective.

- On the other hand you have a song encoded at x kbps that at the end of the day (correct me if I'm wrong) will deliver different kind of frequencies. This element is also objective.

Honestly, I know nothing about ABX test but by definition, some frequencies of the song will fall within the range you can hear, and some won't.

What I'm trying to say, is that despite the fact that I can say that I cannot hear a difference in an ABX test, objectively I must hear it if my ears are able to process the frequencies. Am I so wrong?
[a href="index.php?act=findpost&pid=356605"][{POST_SNAPBACK}][/a]


Yes you are wrong.  Your ears/brain are only processing some information based on a complicated set of rules and their own physical limitations.  Furthermore, while you may be able to hear all frequencies within a band, you may not hear certain frequencies under certain conditions (for instance, very quiet sounds may not be noticed in a noisy room).

If you can hear the difference,  you should eventually be able to ABX the difference given enough time and motivation.

ABX Debate

Reply #18 –
Quote
The debate is simple. Is it okay to rely _soley_ on ABX testing? More technically: If two samples are ABXed and the results indicates that the listener cannot tell the difference, can this be _defined_ as transparent?[a href="index.php?act=findpost&pid=321579"][{POST_SNAPBACK}][/a]


All you can do is say that "in this test, the listener(s) did not make any distinction". No more, no less.
-----
J. D. (jj) Johnston

ABX Debate

Reply #19 –
Quote
All you can do is say that "in this test, the listener(s) did not make any distinction". No more, no less.
no. if n (listeners) is high enough and tests are held under reproducible and standardized circumstances minimizing other variables that might interfere with the results then you can say with a certain amount of confidence (e.g. 95%) that results are valid to the whole population.
Nothing but a Heartache - Since I found my Baby ;)

ABX Debate

Reply #20 –
Since when has Hydrogenaudio become the birthplace of "magical thinking"?  It's kind of like attempting to argue the case of Free Will Vs. Determinism.  I personally just like to settle for Compatiblism and call it a day. 
budding I.T professional

ABX Debate

Reply #21 –
Quote
All you can do is say that "in this test, the listener(s) did not make any distinction". No more, no less.
[a href="index.php?act=findpost&pid=357264"][{POST_SNAPBACK}][/a]
I agree.
Quote
no. if n (listeners) is high enough and tests are held under reproducible and standardized circumstances minimizing other variables that might interfere with the results then you can say with a certain amount of confidence (e.g. 95%) that results are valid to the whole population.
[a href="index.php?act=findpost&pid=357266"][{POST_SNAPBACK}][/a]
I disagree. Because of the structure of ABX testing, the negative case (no difference can be heard) is can be trivially affected by expectation bias. For the positive case, however, it reduces the effect of expectation bias to an insignificant level. For example, If I am comparing 32kbps MP3 versus 44.1/16 wav and I truly expect there not to be a difference, I can bias the result of the test in such a way as a negative result is produced.

Surely, based on this, you can't prove a negative hypothesis with the ABX method. If you think I am wrong, please share your reasons.

ABX Debate

Reply #22 –
Quote
Quote
All you can do is say that "in this test, the listener(s) did not make any distinction". No more, no less.
no. if n (listeners) is high enough and tests are held under reproducible and standardized circumstances minimizing other variables that might interfere with the results then you can say with a certain amount of confidence (e.g. 95%) that results are valid to the whole population.
[a href="index.php?act=findpost&pid=357266"][{POST_SNAPBACK}][/a]



You can always sample population, but you can never make more than a probabilistic statement in either direction, please.
-----
J. D. (jj) Johnston

ABX Debate

Reply #23 –
Quote
Surely, based on this, you can't prove a negative hypothesis with the ABX method. If you think I am wrong, please share your reasons.
[a href="index.php?act=findpost&pid=357284"][{POST_SNAPBACK}][/a]



You can use controls to test the sensitivity of any given test. You can make a probabilistic statement based on any given test. There is no "proof" in any case, either way, only stronger and stronger (as results and 'n' go up) probabilistic proof.

I personally take "proof" to mean what one "proves" in a mathematical proof, and there is no absolute certainty in subjective testing, only probabilistic certainty.
-----
J. D. (jj) Johnston

ABX Debate

Reply #24 –
[a href="index.php?act=findpost&pid=357266"][{POST_SNAPBACK}][/a]
Quote
For example, If I am comparing 32kbps MP3 versus 44.1/16 wav and I truly expect there not to be a difference, I can bias the result of the test in such a way as a negative result is produced.

Does that count as properly performing an ABX test  - deliberately saying the files are indistiguishable when you can actually hear a difference (selecting B when you know the correct answer is A)?

To me that is more akin to making a concerted attempt to bias/sabotage the test. Any experiment will become useless if testers makes a decision to ignore, or say the opposite, to what their senses are telling them.

If a sufficient group of testers perform the same test, it is unlikely that all of them will share the same bias, and thus it is likely that more accurate results will out-weigh what would be later considered an anomaly.

If a hundred people are asked what colour the sky is on a clear day, and I say red, knowing I actually see blue, yet the other 99 people say that it is blue, then we can be a lot more confident the sky is blue, rather than red. The 'bad' data gets filtered out in the end.