Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: MP3 decoders test : MAD isn't so good! (for me...) (Read 92893 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

MP3 decoders test : MAD isn't so good! (for me...)

I tried to compare different mp3 decoders, in order to evaluate the benefits of dithering (MAD and foobar2000) and noise shaping (foobar2000 only), against simplier decoders, as LAME or Fraunhofer (Winamp 5.0). I didn't hope anything first. But I've finally succeed with one sample, and  decided then to resume the test with others samples.
I've wrote a complete review in french, but I can't translate it properly. In few words:


• For this test, I didn't reproduced real listening conditions. Difference are so subtle that I have to increase the level of my amp in order to maximise the audible difference.
• In order to make it possible without being completely deaf in few seconds, I've selected very quiet samples. With classical music, it's not really hard to find such samples (even full tracks are sometimes totally quiet). I didn't evaluate any decoders on pre-instrumental silence, but only when music was playing


Four decoders were tested:

• MAD, basically in order to evaluate the real performance of this mythic decoder, and technically in order to rate the benefits of dithering WITHOUT noise shaping.
• LAME, to mesure the difference between MAD and a simplier decoder.
• Fraunhofer: in order to evaluate possible difference with LAME (both encoders are really similar, so difference may not be audible or existent)
• foobar2000: to see if noise shaping could make a difference with MAD dithering.

Eight samples were introduced:

I've mixed different musical genre: lyrical, orchestral, piano, chamber, electronic. I've tried to mixed dithered and undithered mastered CD too.


CONCLUSIONS:

MAD isn't impressive. On four samples, result was worse than simplest decoders: increase in noise, without audible benefit on other points of the signal (aliasing, higher definition, etc...). But on some other samples, real improvements were noticed (less aliasing, less naturel and less synthetic sound)

LAME and Fraunhofer (WA5) are close, but not totally identical. Both are noisy, but less than MAD (good point). With some samples, sound was irregular, unatural... MAD partailly correct this feeling, and foobar2000 was perfect in comparison. Something like aliasing or moiré was introduced by the decoding stage. Slight difference between LAME and Fraunhofer (hard to ABX : up to 30...50 trials), in favour of Fraunhofer, less synthetic and maybe slightly more noisy

foobar2000 : each time winner. The less noisy of all challengers on all samples. The only additionnal noise audible was a very high-pitched band, very typical, and audible on extreme conditions (near silence sound, and with my amp pump to the maximum). All aliasing, irregularities... were gone: instruments are smooth, naturel. Nothing synthetic at all. With one sample (the #7), the sound was clearly better than the reference file (CD!). The lack of dithering on mastering seem to be prejudicial on very low volume part, and foobar2000 decoding correct the aliasing present on the CD. Simply amazing conclusion....




All results are here :
www.foobar2000.net/mp3decoder
http://www.foobar2000.net/mp3decoder/conclusion.htm
(I've translated in english the summary of my comments).


Comments are welcome.


P.S. It's not a MAD CHALLENGE. I repeat again that the listening conditions of the test are not reproductible on daily music listening. I've just put the decoder behind a magnifying glass, by using low volume samples.

MP3 decoders test : MAD isn't so good! (for me...)

Reply #1
I suppose you tested with ABC/HR blind rating although you don't say it in the message?

Your pages here would be fantastic but too bad you chose to use french.. 

The "better than CD" sentence is maybe a bit controversial though, but I understand what you are after... 
Juha Laaksonheimo

MP3 decoders test : MAD isn't so good! (for me...)

Reply #2
ABC/HR, yes (the log files are available on the site, or here.Of course, the notation or rank can't be the same with this test than for a 64 kbps listening test. Difference between different decoders was magnified, as well as samples were magnified by increasing the listening volume.

For french language, ask my parents or english teachers: I didn't choose my country

MP3 decoders test : MAD isn't so good! (for me...)

Reply #3
Impressive. Thanks for your effort. Several people before have noted that MAD's dithering is "sub-optimal" and inferior to fb2k's noiseshaped dithering.
"To understand me, you'll have to swallow a world." Or maybe your words.


MP3 decoders test : MAD isn't so good! (for me...)

Reply #5
One "problem" with Guru's interesting results are that those are presented on www.foobar2000.net. Not very unbiased sounding place despite of all the blind ratings and ABX -results..
Juha Laaksonheimo

MP3 decoders test : MAD isn't so good! (for me...)

Reply #6
I agree 



MP3 decoders test : MAD isn't so good! (for me...)

Reply #7
I'm amazed (but very glad) that you've managed to subjectively confirm my old objective test results: MAD was/is measurably inferior to the best.

http://mp3decoders.mp3-tech.org/24bit.html
(compare the various MAD compiles with l3dec)

I felt quite harsh for pointing this out at the time. I assumed no one would ever hear the difference with real music. It's interesting that it can be audible (in extreme conditions). FWIW the differences in dither algorithms could be greater with a very clean 24-bit source. (you state you used CDs)

It also suggests that the assumption which runs through the foobar2k GUI (which seems to be "this program performs very accurate processing, but it's probably impossible to hear the difference") is too modest!

Anyway, I wish I had your ears guruboolez!

Cheers,
David.

MP3 decoders test : MAD isn't so good! (for me...)

Reply #8
Quote
Anyway, I wish I had your ears guruboolez!

Golden ears would probably perceive a difference between different decoders on real listening conditions. Common ears, like mine, need to push the volume to extreme position. Try, please... I was the first to be surprise to hear differences.

At the beginning, I mainly heard different noise level... Then, I was able to ditinguish different things (that may explain why, at the end of the test, notation between decoders was more distant).

I've uploaded the lossless samples (which are -funny things- clearly inferior to the 320 CBR encoding filesize) in optimfrog format, so everybody could make the same test.

MP3 decoders test : MAD isn't so good! (for me...)

Reply #9
Good thorough test

Thanks for taking the time to do this... and all the other things you've contributed.

MP3 decoders test : MAD isn't so good! (for me...)

Reply #10
Quote
Golden ears would probably perceive a difference between different decoders on real listening conditions.

You are so modest, but this is pretty amazing stuff.

Thank you.
I'm the one in the picture, sitting on a giant cabbage in Mexico, circa 1978.
Reseñas de Rock en Español: www.estadogeneral.com

MP3 decoders test : MAD isn't so good! (for me...)

Reply #11
hmm. maybe it should be added that foobar is not the decoder itself. it uses the mpg123 library for decoding which is available for Winamp, too afaik. So the separation Winamp vs foobar is not correct and looks even more strange, as this test is provided on the foobar website 

better would be: MAD vs LAME vs Fraunhoffer vs Mpeg123

MP3 decoders test : MAD isn't so good! (for me...)

Reply #12
Yes, and no.
I didn't test mpg123 alone, because there was dithering/noise shaping... from foobar2000. The most interessing thing to note is that mp4, vorbis, mpc... playback will profit from the same technique.

I explained that in french : the test isn't an audio player comparison (mainly because audio players are not only decoders). The test is not really a mp3 decoder test, but more a decoding technique test : dithered vs non-dithered decodings and noise-shaped dithering against simple dithering. LAME and Fraunhofer, sharing the same principles, are really close, if not the same on three or four samples. Dithering introduced a real difference (not always good, because MAD files were systematically the easiest to ABX), and noise shaping other differences.

I wonder why other audio players didn't use these techniques. Is it really hard to program? Dithering/noise shaping is very common on studio mastering: most of my CDs have visible noise shaping. So if studios use them, it must be good.

MP3 decoders test : MAD isn't so good! (for me...)

Reply #13
But are you sure that these improvements in sound quallity are the benefit of the dithering/noise shaping or the benefit of the decoder library itself. Maybe the same increase in quality could be heard by using the in_mpg123 plugin in winamp, too.

I would rather test Winamp in_mpg123 vs Foobar mpg123, so you can be sure that the dithering and noise shaping was the determining factor in increasing the quality. Now it can be the decoder itself.

MP3 decoders test : MAD isn't so good! (for me...)

Reply #14
Don't other players just do the obvious, simple thing: decode to 16-bits and truncate?

You can't noise shape and dither unless you have more than 16-bits available to start with. It must be there inside the decoder (well, at least partly, or at least in some decoders?) - but foobar is the first player where all these issues have been properly thought through.

Don't forget just how bad Winamp has been at times during its history. The old Nitrane decoder was barely 14-bit accurate at times - no point dithering it!


It seems that some players (did or do) think it's enough just to get something resembling music out of an mp3 - the quality wasn't/isn't important.

Cheers,
David.

P.S. surely mpg123 and lame give basically the same decodes (with occasional LSB differences), and all FhG decoders are likewise similar (with occasional LSB differences). FhG vs lame (=mpg123?) gives continuous LSB differences; whats more, the lame decoder apparently includes -90dB harmonic distortion when decoding to 16-bit - or at least it did when I tested it years ago.

MP3 decoders test : MAD isn't so good! (for me...)

Reply #15
huh? I cannot really follow your post.

I don't think the Shibatch mpg123 library and lame are related in any kind. 16+ bit output should be possible with winamp mpg123 plugin as well.

Are you confusing something?

MP3 decoders test : MAD isn't so good! (for me...)

Reply #16
Oh damn! I've always thought dithering was no good.
Liberate tutemae ex inferis

MP3 decoders test : MAD isn't so good! (for me...)

Reply #17
Quote
huh? I cannot really follow your post.

I don't think the Shibatch mpg123 library and lame are related in any kind. 16+ bit output should be possible with winamp mpg123 plugin as well.

Are you confusing something?

http://lame.sourceforge.net/

claims that

"LAME 3.xx uses software from:
Decoding engine (for the frame analyzer):  mpglib, from the mpg123 project."

I've never investigated the details of this statement, but the lame output and the Winamp mpg123 output are amazingly similar.

Cheers,
David.

MP3 decoders test : MAD isn't so good! (for me...)

Reply #18
they adopted and improved hipp's mpglib and integrate for decoding

...and also made a hip interface (libmp3hip) for a straight decoding library


later

MP3 decoders test : MAD isn't so good! (for me...)

Reply #19
oops, then the fault was on my side. Good to know, though.

If you are correct, these conclusons can be drawn from the test:
- Neither MAD. nor mpg123, nor FhG can outperform its competitors (one better here, the other better there)
- only a combination with dithering and noise shaping will offer signifivantly better quality.

Right?

MP3 decoders test : MAD isn't so good! (for me...)

Reply #20
Quote
Oh damn! I've always thought dithering was no good.

oh no, dithering is very important, especially when you reduce the resolution from 24 bits to 16 bits. it's consecutive re-dithering which is a questionable practice, especially when the source is only 16 bits. however, if we use mp3s as an example, if the bitrate is high enough, it may be possible to retrieve data above 16 bits, thus preserving more ambiance and detail which would surely be lost if we were to truncate at 16 bits. there is no guarantee, however, that you will hear a lot of difference or any difference at all. the side effect of re-dithering is more noise in the audio signal but this can be somewhat offset with proper noise shaping, although this is very subjective.
Be healthy, be kind, grow rich and prosper

MP3 decoders test : MAD isn't so good! (for me...)

Reply #21
Quote
the side effect of re-dithering is more noise in the audio signal but this can be somewhat offset with proper noise shaping, although this is very subjective.


If the original dithering was noise shaped, it was mainly removed during mp3 encoding because of the applied lowpass filter.

MP3 decoders test : MAD isn't so good! (for me...)

Reply #22
Quote
oops, then the fault was on my side. Good to know, though.

If you are correct, these conclusons can be drawn from the test:
- Neither MAD. nor mpg123, nor FhG can outperform its competitors (one better here, the other better there)
- only a combination with dithering and noise shaping will offer signifivantly better quality.

Right?

Wrong.

Read this quote from an earlier post from guruboolez on this thread again:
Quote
Golden ears would probably perceive a difference between different decoders on real listening conditions. Common ears, like mine, need to push the volume to extreme position. Try, please... I was the first to be surprise to hear differences.


There are no significant differences of any kind.
I'm the one in the picture, sitting on a giant cabbage in Mexico, circa 1978.
Reseñas de Rock en Español: www.estadogeneral.com

MP3 decoders test : MAD isn't so good! (for me...)

Reply #23
I've done a similar test (~ 1 year ago I guess, but I can't find where I posted it  ):
  • Take some mp3s (3.90.2 --alt-preset standard IIRC)
  • Use mp3directcut to apply fadeouts (0 -> -40 dB IIRC)
  • Decode using foobar2000, winamp, xmplay
  • Apply fadeins opposite to the mp3directcut fadeouts (e.g. 0 -> +40 dB)
  • Listen + compare
IIRC the results were these:
- With Winamp (native decoder, not MAD) there was clearly audible truncation distortion
- With foobar2000 (I can't remember what dither type I used - I don't even know if there were options to choose) there was no distortion at all but a clearly audible noise floor (of course exagerated by amplification)
- XMplay's decoder was some kind of compromise: Less distortion than Winamp and lower audible noise level than fb2k

This way a similar test can be performed without need for very good (=low noise floor) equipment. It's still hard to tell how these results are related to listening under real life conditions though.

Edit: Seems like I never posted about the test unfortunately. Some time before (I think even before I knew foobar2000) I did the same test XMPlay (2.5) vs. MAD, the result is here
Later, I repeated it with Winamp 2.8x, fb2k 0.dunno and XMPlay 2.6 as described above - well, if anyone's really interested, it should be very easy to repeat ....
Let's suppose that rain washes out a picnic. Who is feeling negative? The rain? Or YOU? What's causing the negative feeling? The rain or your reaction? - Anthony De Mello

MP3 decoders test : MAD isn't so good! (for me...)

Reply #24
Quote
I wonder why other audio players didn't use these techniques. Is it really hard to program? Dithering/noise shaping is very common on studio mastering: most of my CDs have visible noise shaping. So if studios use them, it must be good.

They slow down the player for something which doesn't have a benefit in casual listening. This is why it defaults to off in foobar too.