IPB

Welcome Guest ( Log In | Register )

3 Pages V   1 2 3 >  
Reply to this topicStart new topic
What does ABX really prove?
audioslut512
post Nov 10 2007, 18:59
Post #1





Group: Members
Posts: 19
Joined: 8-November 07
Member No.: 48571



I recently did some ABX tests of music encoded in FLAC levels 0-8.

To my surprise I could tell the difference more often than not.

Yet I couldn't help but wonder, what does this really prove?

It seems that if an audible difference exists between two tracks, that difference wouldn't necessarily indicate anything as to which track is of higher quality, but rather, that those two tracks are merely different from one another...

Am I missing the point in all of this?
Go to the top of the page
+Quote Post
user
post Nov 10 2007, 19:05
Post #2





Group: Members
Posts: 873
Joined: 12-October 01
From: the great wide open
Member No.: 277



Please show details of your abx tests.
Do you have logs ?

And more important,

describe more exactly, what do you have abxed ?

flac -0 vs. flac -8 of the same wav source file ?

well, if you should have proven to listen a difference between these files, then you might suffer at strong imagination or your abx test was not blind, but nevertheless don#t conclude from your personal test to ABX tests.
it depends always on the specific setup, what an abx test can prove.


--------------------
www.High-Quality.ch.vu -- High Quality Audio Archiving Tutorials
Go to the top of the page
+Quote Post
tgoose
post Nov 10 2007, 19:06
Post #3





Group: Members
Posts: 407
Joined: 12-April 05
Member No.: 21399



An ABX test tells you how likely it is that you can tell the difference between two tracks. If by "more often than not" you mean that you got it right more than 50% of the time, that doesn't prove anything. If you get significantly above 50% over a large number of trials (more than 20) and it's repeatable then that might imply that there is something wrong with the FLAC encoder. Do you have the ABX log saved to post so I can see what's happened?

And yes, you're right, ABX can never tell you which is "better", only that there is a difference. So it's only useful for tuning something for transparency or to test for problems with encoding, not really many other things.
Go to the top of the page
+Quote Post
maksm
post Nov 10 2007, 19:07
Post #4





Group: Members
Posts: 2
Joined: 3-October 07
Member No.: 47553



No, you're not.
If there is a difference, and the only source of differences is an encoding (because we encoded the same track), than it must be the encoding.
And we "know" (this isn't neceseraily true) that the encodings have a (mathematical) scale of their quality.
Therefore the lower quality one must be "worse".

This post has been edited by maksm: Nov 10 2007, 19:09
Go to the top of the page
+Quote Post
tgoose
post Nov 10 2007, 19:11
Post #5





Group: Members
Posts: 407
Joined: 12-April 05
Member No.: 21399



QUOTE (maksm @ Nov 10 2007, 18:07) *
No, you're not.
If there is a difference, and the only source of differences is an encoding (because we encoded the same track), than it must be the encoding.
And we "know" (this isn't neceseraily true) that the encodings have a (mathematical) scale of their quality.
Therefore the lower quality one must be "worse".


No?

If you had a poorly tuned 320kbps MP3 and a well tuned VBR MP3 from a different encoder, and you ABXed the two then if you could tell a difference there's not necessarily a way of telling which is better without referencing the original uncompressed version.
Go to the top of the page
+Quote Post
greynol
post Nov 10 2007, 19:15
Post #6





Group: Super Moderator
Posts: 10009
Joined: 1-April 04
From: San Francisco
Member No.: 13167



There are other tests to rank differences. The point of ABX is to determine if you can distinguish a difference.


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
Nikaki
post Nov 10 2007, 19:20
Post #7





Group: Members
Posts: 100
Joined: 6-March 07
Member No.: 41226



Doing ABX on FLAC and actually being successful in it, while a binary comparison of the resulting wav files shows no difference, is an indication that you're imagining things biggrin.gif
Go to the top of the page
+Quote Post
audioslut512
post Nov 10 2007, 19:36
Post #8





Group: Members
Posts: 19
Joined: 8-November 07
Member No.: 48571



QUOTE (greynol @ Nov 10 2007, 13:15) *
There are other tests to rank differences. The point of ABX is to determine if you can distinguish a difference.


This is more along the lines of what I was asking. It seems that ABX only helps you determine whether there is or is not a difference between two tracks encoded from the same source, and not necessarily whether those differences make one track of higher quality than another...
Go to the top of the page
+Quote Post
greynol
post Nov 10 2007, 19:40
Post #9





Group: Super Moderator
Posts: 10009
Joined: 1-April 04
From: San Francisco
Member No.: 13167



An ABX test cannot prove that there isn't a difference. It only has the ability to demonstrate that the specific person taking the test was able to determine that there was a difference. These are not the same thing.

This post has been edited by greynol: Nov 10 2007, 19:44


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
audioslut512
post Nov 10 2007, 19:43
Post #10





Group: Members
Posts: 19
Joined: 8-November 07
Member No.: 48571



QUOTE (greynol @ Nov 10 2007, 13:40) *
Actually, an ABX test cannot prove that there isn't a difference, it only has the ability to demonstrate that the specific person taking the test was able to determine there was a difference.


Agreed.
Go to the top of the page
+Quote Post
greynol
post Nov 10 2007, 19:50
Post #11





Group: Super Moderator
Posts: 10009
Joined: 1-April 04
From: San Francisco
Member No.: 13167



I concur with the first response you were given, btw. Telling us that you can tell an audible difference between flac compression levels brings your methodology into serious question.


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
audioslut512
post Nov 10 2007, 20:03
Post #12





Group: Members
Posts: 19
Joined: 8-November 07
Member No.: 48571



QUOTE (greynol @ Nov 10 2007, 13:50) *
I concur with the first response you were given, btw. Telling us that you can tell an audible difference between flac compression levels brings your methodology into serious question.



To clarify, I didn't mean to say I could tell the difference between FLAC 0-8 more often than not...

Rather, that of some of the tracks that I tested, which I had previously encoded from the same source (using the same FLAC 1.2.1 codec), difference to my own ear was indicated to me (and only me) by a probability that was statistically significant.

HOWEVER, I did NOT mean to start some sort of argument as to the means by which I performed the tests or for that matter, anything to do with my testing at all (I probably shouldn't have even mentioned those tests to begin with, it was just for context).

I was just trying to frame a much more general question, pertaining to the difference between two tracks sounding different as opposed to one track sounding better than another...

My apologies if I came across otherwise!
Go to the top of the page
+Quote Post
greynol
post Nov 10 2007, 20:08
Post #13





Group: Super Moderator
Posts: 10009
Joined: 1-April 04
From: San Francisco
Member No.: 13167



You're not understanding.

If you think you're able to tell the difference between two flac encodings and feel that the difference lies only in the encodings themselves, then you aren't conducting a proper ABX test.


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
saratoga
post Nov 10 2007, 20:08
Post #14





Group: Members
Posts: 5000
Joined: 2-September 02
Member No.: 3264



QUOTE (audioslut512 @ Nov 10 2007, 12:03) *
To my surprise I could tell the difference more often than not.


QUOTE (audioslut512 @ Nov 10 2007, 12:03) *
To clarify, I didn't mean to say I could tell the difference between FLAC 0-8 more often than not...


QUOTE (audioslut512 @ Nov 10 2007, 12:03) *
difference to my own ear was indicated to me (and only me) by a probability that was statistically significant.


Wait, so which is it?
Go to the top of the page
+Quote Post
Cosmo
post Nov 10 2007, 20:31
Post #15





Group: Members
Posts: 913
Joined: 10-January 05
Member No.: 18979



QUOTE (audioslut512 @ Nov 10 2007, 12:59) *
It seems that if an audible difference exists between two tracks, that difference wouldn't necessarily indicate anything as to which track is of higher quality, but rather, that those two tracks are merely different from one another...

This is absolutely true.

But you shouldn't ever hear a difference between two lossless files, because regardless of the type or amount of compression, they are always decoded (bit identical to original) at playback. So if a difference IS audible, it must be due to improper test methodology or buggy software or somesuch reason...

This post has been edited by Cosmo: Nov 10 2007, 20:33
Go to the top of the page
+Quote Post
audioslut512
post Nov 10 2007, 23:47
Post #16





Group: Members
Posts: 19
Joined: 8-November 07
Member No.: 48571



QUOTE (Mike Giacomelli @ Nov 10 2007, 14:08) *
QUOTE (audioslut512 @ Nov 10 2007, 12:03) *

To my surprise I could tell the difference more often than not.


QUOTE (audioslut512 @ Nov 10 2007, 12:03) *
To clarify, I didn't mean to say I could tell the difference between FLAC 0-8 more often than not...


QUOTE (audioslut512 @ Nov 10 2007, 12:03) *
difference to my own ear was indicated to me (and only me) by a probability that was statistically significant.


Wait, so which is it?


The ABX tests which I performed were on tracks which were contained within the set: FLAC levels 0-8.

If I performed an ABX test, the files tested were limited to FLAC 1,2,3,4,5,6,7,8.

That does NOT indicate that I tested ALL levels 0-8.

OF THE TRACKS WHICH I DID TEST, I was able to discern a difference more often than not.

Statistically significant is that which is unlikely to occur by chance.

A statistically significant difference in this case, merely implies that statistical evidence indicates that a discernible difference has a greater probability of existing than it does a probability of not existing.

Whether that difference is due to errors in encoding, playback, or whatever, is entirely irrelevant.
Go to the top of the page
+Quote Post
eevan
post Nov 11 2007, 00:17
Post #17





Group: Members
Posts: 537
Joined: 9-April 07
From: Belgrade, Serbia
Member No.: 42357



No offence, but having read all your posts, I'm not sure exactly what are you implying? Why should this be of any interest if you say that it's irrelevant to pinpoint the cause of your test result.


--------------------
If age or weaknes doe prohibyte bloudletting you must use boxing
Go to the top of the page
+Quote Post
AndyH-ha
post Nov 11 2007, 00:31
Post #18





Group: Members
Posts: 2210
Joined: 31-August 05
Member No.: 24222



In the laboratory, and in most audiophile living rooms, people can be psyched into discerning a difference where none exists. The only thing that accounts for that is belief or expectation creating abnormal conditions inside the listener's head. A statistically significant score, even one approaching certainty, does not make any actual difference come into existence.

FLAC encoding/decoding can be shown to produce zero differences, and often has been so shown, both mathematically and empirically. Proper ABX tests disallow any possibility of expectation or belief effecting test scores. With no data differences, and no opportunity for perceptual bias, any indicated differences, no matter what the test score, indicates something wrong, something not normal in the data or the process.
Go to the top of the page
+Quote Post
greynol
post Nov 11 2007, 00:39
Post #19





Group: Super Moderator
Posts: 10009
Joined: 1-April 04
From: San Francisco
Member No.: 13167



What an incredible waste of time!

I propose that this guy submit samples and an ABX log indicating that he can distinguish a difference or the thread be closed.

audioslut512:
It is universally accepted that foobar2000 is able to perform proper ABX tests. Make sure you use it and spare us the fluffy language about statistical evidence which is ringing quite hollow.

http://www.hydrogenaudio.org/forums/index....showtopic=16295

This post has been edited by greynol: Nov 11 2007, 00:43


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
odyssey
post Nov 11 2007, 00:51
Post #20





Group: Members
Posts: 2296
Joined: 18-May 03
From: Denmark
Member No.: 6695



How can you discuss this? blink.gif

QUOTE (audioslut512 @ Nov 10 2007, 19:59) *
Am I missing the point in all of this?

YES! Here is why:
QUOTE (audioslut512 @ Nov 10 2007, 19:59) *
I recently did some ABX tests of music encoded in FLAC levels 0-8.

You can't DO that! FLAC is a lossless format - That means the source are NOT degraded, no matter which level of compression you choose. The only differences the levels of a lossless encoder produce, is encoding and decoding speeds.

If you try to ABX these, you are actually comparing the same stuff, YES that's why you can't distinguish them.

You will usually use ABX tests to determine if you are able to distinguish a lossy encoded file against the original. It's very useful to make people shut up when they believe they can tell high bitrate encoded lossy files from the source tongue.gif


--------------------
Can't wait for a HD-AAC encoder :P
Go to the top of the page
+Quote Post
kwanbis
post Nov 11 2007, 01:11
Post #21





Group: Developer (Donating)
Posts: 2362
Joined: 28-June 02
From: Argentina
Member No.: 2425



QUOTE (audioslut512 @ Nov 10 2007, 17:59) *
I recently did some ABX tests of music encoded in FLAC levels 0-8.

You know that FLAC is lossless right? It means that no matter what the compression level, you always get the same "sound", the only diference is the size of the file.

What you are saying is like saying you compressed a TXT file with ZIP at two compression levels, and when you uncompressed them, they where diferent.


--------------------
MAREO: http://www.webearce.com.ar
Go to the top of the page
+Quote Post
guruboolez
post Nov 11 2007, 01:29
Post #22





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (audioslut512 @ Nov 11 2007, 00:47) *
OF THE TRACKS WHICH I DID TEST, I was able to discern a difference more often than not.

Statistically significant is that which is unlikely to occur by chance.


May I ask you the score? More often than not is not very precise. I can (and everybody should achive the same performance) also reach some 55% or 60% good trials while ABXing lossless from time to time. It also work with WAV vs WAV ABX test and even without listening to anything. You just need to be a bit lucky during the first trials and limit them to ~20...30, not more. The overall score will naturally tend to 50% on long term but if you only keep the first ones you may sometime get a probablity higher than 50% and maybe the illusion of success.

That's why I'd like to have some precisions: what are the score of all tests (not only the most favorable session)? Did you fix the number of trials before starting?

This post has been edited by guruboolez: Nov 11 2007, 01:31
Go to the top of the page
+Quote Post
audioslut512
post Nov 11 2007, 02:59
Post #23





Group: Members
Posts: 19
Joined: 8-November 07
Member No.: 48571



just forget it kids.
Go to the top of the page
+Quote Post
saratoga
post Nov 11 2007, 03:16
Post #24





Group: Members
Posts: 5000
Joined: 2-September 02
Member No.: 3264



I still don't really understand what you did. Could you just post the logs?
Go to the top of the page
+Quote Post
audioslut512
post Nov 11 2007, 04:56
Post #25





Group: Members
Posts: 19
Joined: 8-November 07
Member No.: 48571



QUOTE (Mike Giacomelli @ Nov 10 2007, 21:16) *
I still don't really understand what you did. Could you just post the logs?



I would if I had them. All I did was ABX Radiohead's Kid A, encoded in FLAC level 4 and level 8, both directly from the CD. I matched the correct tracks to one another 9 of the 10 times I ran it. That's all. Maybe I screwed something up while encoding them that caused one to sound different from the other. Ultimately I had no idea which one was which, I just managed to notice the differences and match the tracks accordingly. Hence my post...
Go to the top of the page
+Quote Post

3 Pages V   1 2 3 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 1st October 2014 - 16:10