Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Mad challenge - my result (Read 22254 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Mad challenge - my result

I've tryed the Mad vs FhG challenge at 16 bits. (quite hard with this bad integrated sound chip):

  0 of  1, p = 1.000
  1 of  2, p = 0.750
  2 of  3, p = 0.500
  3 of  4, p = 0.313
  4 of  5, p = 0.188
  4 of  6, p = 0.344
  5 of  7, p = 0.227
  6 of  8, p = 0.145
  7 of  9, p = 0.090
  8 of  10, p = 0.055
  8 of  11, p = 0.113
  8 of  12, p = 0.194
  8 of  13, p = 0.291
  8 of  14, p = 0.395
  9 of  15, p = 0.304
10 of  16, p = 0.227
10 of  17, p = 0.315
11 of  18, p = 0.240
11 of  19, p = 0.324
12 of  20, p = 0.252
13 of  21, p = 0.192
13 of  22, p = 0.262
14 of  23, p = 0.202
15 of  24, p = 0.154
16 of  25, p = 0.115
17 of  26, p = 0.084

Mad challenge - my result

Reply #1 –
sorry gabriel. what do these values tell me?

Mad challenge - my result

Reply #2 –
Quote
sorry gabriel. what do these values tell me?

I believe this means that he was successfully able to ABX FhG's MP3 decoder from the MAD decoder 17 out of 26 times.

Edit: Changed quote date and time to ISO 8601 format.

Mad challenge - my result

Reply #3 –
Quote
Quote
sorry gabriel. what do these values tell me?

I believe this means that he was successfully able to ABX FhG's MP3 decoder from the MAD decoder 17 out of 26 times.

...or MAD from FhG... maybe you could clear things up Gabriel 
Nothing but a Heartache - Since I found my Baby ;)

Mad challenge - my result

Reply #4 –
Usually, a p <= 0.05 is considered a significant result.  This is pretty close though, which to me indicates that there is a very good chance that more testing would result in a statistically significant result.
I am *expanding!*  It is so much *squishy* to *smell* you!  *Campers* are the best!  I have *anticipation* and then what?  Better parties in *the middle* for sure.
http://www.phong.org/

Mad challenge - my result

Reply #5 –
Gabriel, Which Version is The MAD? 

Downloads? 
It's My Life,
It's Now Or Never,
I Ain't Gonna Live Forever,
I Just Want To Live While I'm Alive.

Mad challenge - my result

Reply #6 –
Quote
Usually, a p <= 0.05 is considered a significant result.  This is pretty close though

yes, it would be intersting to see some further results.
AFAIK a p-value of 0,01 is more generally accepted, 0,05 is little more like 'bending' the rules (though it is also widly used in tests).
care to do some further testing Gabriel?
Nothing but a Heartache - Since I found my Baby ;)

Mad challenge - my result

Reply #8 –
Quote
...or MAD from FhG... maybe you could clear things up Gabriel

I do not see any difference between discerning FhG from Mad or Mad from FhG.


Quote
Which Version is The MAD?

Mad 0.14.2b
FhG decoder 2.91


I think that it will be hard for me to do better with this sound card. When you push up the volume, there is a lot of noise.

Mad challenge - my result

Reply #9 –
Quote
Quote
...or MAD from FhG... maybe you could clear things up Gabriel

I do not see any difference between discerning FhG from Mad or Mad from FhG.

there's no differnece the way you say it. however, I just wanted to know which of the two represents the numbers in the end. did you mange to put your finger on MAD or Fhg 17 times of 26? sorry if I didn't make that clear.
Nothing but a Heartache - Since I found my Baby ;)

Mad challenge - my result

Reply #10 –
Quote
Quote
Quote
...or MAD from FhG... maybe you could clear things up Gabriel

I do not see any difference between discerning FhG from Mad or Mad from FhG.

there's no differnece the way you say it. however, I just wanted to know which of the two represents the numbers in the end. did you mange to put your finger on MAD or Fhg 17 times of 26? sorry if I didn't make that clear.

You are missing the point in an ABX-test. An unknown sample X (which is either A or B, randomly chosen at each trial) is compared to two references (A and B ). The tester has to decide if "X is the same as A" or if "X is the same as B" is true.

Out of 26 trials taken, 17 times the answer was correct. This is not connected to whether A is MAD or Fhg.
 

Mad challenge - my result

Reply #11 –
right.
and in order to tell the difference, one file have to be worse quality (or sound somehow differnet to you) than the other, am I right? otherwise you could not tell a difference...
Nothing but a Heartache - Since I found my Baby ;)

Mad challenge - my result

Reply #12 –
Yes, but the difference relation is symmetric.
(A /= B is equivalent to B /= A)

Is your question, what sounded better to him?

Mad challenge - my result

Reply #13 –
exactly 
Nothing but a Heartache - Since I found my Baby ;)

Mad challenge - my result

Reply #14 –
Quote
Is your question, what sounded better to him?

None sounded better, just slightly different.

You have to keep in mind that I did not achieved 99% confidence, but only 91.6% (94.5 at max).
It means that you can not really considere it as a definitive answer to the Mad vs FhG challenge. It gives a positive indication, that is all.

Mad challenge - my result

Reply #15 –
I just did a compliance test for both MAD 0.15.0 and the decoder in Winamp (latest). Result is that for the comliance sine sweep MAD (without dithering) reaches 23 bit accuracy and Winamp 16 (winamp output was 16 bit, so that's the maximum). However on the non-sine sweep reference files the Winamp decoder was a lot less accurate ("only" 15 bit accuracy), while MAD kept 21 or 22 bit accuracy. Except for 1 file where both decoders reached an accuracy of 0 bit (1kHz 0dB sine).

However (!!!), when I tried the MAD plugin for winamp (0.14.2), with 16 bit output, it reached only 14 bits accuracy on all files except the one 0 bit file (and 16 bit on the compliance sine sweep).

So apparantly the difference is quite big. FhG screws up but MAD screws up even more. Does this MAD Winamp plugin do dithering? And yes, my guess is that for some people this difference is big enough to be audible.

Menno

edit:
Note: command line MAD with 16 bit output and no dithering produced 15 to 16 bit accuracy except for the 1kHz sine.

Mad challenge - my result

Reply #16 –
BTW: Foobar did not reach more than 18 bit accuracy on 24 bit undithered output (however still fully 16 bit compliant), but when using 16 bit undithered output it gave a result of 15 bit accuracy for most files (including the compliance sine sweep) and 16 bit on 2 files. On that 0dB 1kHz sine wave it got 14 bit (!!).

Menno

Mad challenge - my result

Reply #17 –
Quote
FhG screws up but MAD screws up even more. Does this MAD Winamp plugin do dithering?


Yes, the winamp plugin should do dithering. This is why I think that it should only reach 15bits accuracy, as the lowest bit is dithered. However, there is still 1 bit missing.

Mad challenge - my result

Reply #18 –
Oh wait, I made a mistake with the FhG delay on that 1kHz file, it produces 15 bit accuracy there.

Menno

Mad challenge - my result

Reply #19 –
Quote
Quote
FhG screws up but MAD screws up even more. Does this MAD Winamp plugin do dithering?


Yes, the winamp plugin should do dithering. This is why I think that it should only reach 15bits accuracy, as the lowest bit is dithered. However, there is still 1 bit missing.

Hmm, so it is 15 bit like the undithered output, but loses 1 because of the dithering -> 14.

FhG and MAD only have 16 bit accurate output on the compliance sine sweep. But then again, this is where the compliance is based on, but it is "recommended" that the other files also produce 16 bit accurate output.

I find it strange that MAD produces 22/23 bit accurate results when decoding to 24 bit but produces only 15 bit results on those files when decoding to 16 bit. Must be something with the rounding.

Menno

Mad challenge - my result

Reply #20 –
Quote
You have to keep in mind that I did not achieved 99% confidence, but only 91.6% (94.5 at max).

Also, you didn't seem to have fixed the number of trials before starting. This way (testing until the score is good enough) leads to an overestimation of the confidence, because the result is biased towards a good score.
For example if you had chosen to stop at 16 to begin with, your result would have been 77.3 % only. This must be taken into account in the final estimation.
But it is so complicated that we prefer to fix the number of trials before.
Note that you can still change the number of trials during the test as long as you have not seen your previous results.

Mad challenge - my result

Reply #21 –
Quote
Hmm, so it is 15 bit like the undithered output, but loses 1 because of the dithering -> 14.

In theory, using flat dither with as much as 1 bit of amplitude, dither would only have an effect on the 16th bit, so the 15th bit wout not be touched.

Edit: But if some kind of noiseshaping dither is used, it could have an effect on higher-level bits too (bits < 16).

Mad challenge - my result

Reply #22 –
A bit off topic:

Is the MAD Winamp plugin developed further these days? It's still 0.14.2b and is lacking updates for some time. AFAIK it doesn't support the extended metadata interface used by Winamp Library yet (so working with current 2.9+ Winamp versions isn't very convenient).

Who has any infos?


(sorry for the bit off topic, amano)