Personal multiformat listening test at ~130 kbps, based on classical (baroque) music only
Personal multiformat listening test at ~130 kbps, based on classical (baroque) music only
Oct 11 2003, 16:31
Group: Members (Donating)
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420
During a sleepless night, I decided to perform my own listening test, with samples based on my musical tastes. I chose a friendly bitrate encoding (~130 kbps) : in my opinion, it’s a good compromise for limited hardware space (notebook) and good quality (my preliminary experience with lower bitrate was catastrophic). Of course, testing encoding at this bitrate range is much easier. I also had in mind the idea to build a comfortable library for my notebook, and the idea to clarify some feelings I had about different encoders, on my music only. Among them :
- I suspected lame mp3 to be a good audio format on some (quiet) situation, and to not suffering as much as with pop, metal or electronic music.
- I suspected Nero AAC to be simply worse than Lame MP3 ; I did some encodings on common music the last times, and I was disappointed by the poor results I obtained for a VBR setting (-streaming). I couldn’t conclude on Nero worse quality based on three or four samples. This extend test will be more conclusive.
- I suspected QuickTime AAC and WMA 9 PRO to reach a near-transparency quality at this bitrate.
Unfortunately, I worked with a limited choice of samples. With the start of the new university year, I had to leave my summer house. My discs are not for the moment in my new apartment : only the few one I bought since September. I choose samples from them, and completed the gallery with samples I burned on CD-R some times ago. Most of them are music from baroque period. It’s my favorite one. Therefore, I couldn’t materially built a coherent sample library for this test with romantic or contemporary music. That’s why there’s no piano, no Beethoven, no electronic samples here. That’s why, too, so many harpsichord (hard-to-encode instrument) is present, during singing, sacred, chamber or orchestral music. I did my best to find some samples without this instrument : more of the half.
Therefore, this test only had a limited impact: results are useful for baroque music only ; they may be different for other kind of “classical” ; and they must be totally different for jazz, country, R&B or anything else. Of course, notation is mine, dependant on my own subjectivity, my on hardware setting, and on the mood I had during the test.
I encoded them with foobar2000 (except for WMA9 and PsyTEL AAC). All encodings were replaygained, original too. Decoding stage include the calculated gain (without dithering – maybe not the best choice). I didn’t make any offset correction (needed for some AAC encoding).
Challengers are :
• AAC Faac (don’t know the exact version : I used Case diskwriter component bundled with foobar2000 0.7.1 beta 4), quantizer 128, LC, m/s coding and with TNS.
• AAC Nero (aac.dll 18.104.22.168 & aacenc32.dll 22.214.171.124) : -streaming, LC, High quality
• AAC PsyTEL 2.15 –streaming [ADDED ONE WEEK LATER: I did a big mistake, and placed the good aacenc32.dll in the wrong location Therefore, I tested an old encoder : 126.96.36.199. Some odd artifacts (Dorilla, Passacaglia) are not present in 188.8.131.52. See here for complete report and additional test]
• AAC QuickTime 6.3, CBR 128, High Quality
All AAC files were decoded with faad2 (foobar2000)
• MP3 Lame 3.90.3 –alt-preset 134
• MPC 1.14 –quality 4 –xlevel
• OGG vorbis GT3b1 –q 4.25
• WMA 9 PRO VBR-2pass 128 kbps
(I chose abr 134 in order to be close to 128 kbps, because ABR mode isn’t reliable with most classical stuff. I used –b 4.25 for Vorbis, according to the average size collected some times ago, before Roberto’s test: it was a small mistake).
DECODING & PLAYBACK
All files (including reference) were replaygained with foobar2000, and then decoded with calculated gain and without dithering, No offset correction was applied (needed for correcting some AAC encoding offset) : I didn’t have the correct value.
Note than one sample (« Laudate pueri ») wasn’t properly encode/decode with PsyTEL : 15 seconds are missing, due to a bug (« negative scalfactor… »). I rated the quality for the first part only.
Comparisons were made with ff123 tool (ABC/HR). I didn’t take the time for ABXing (except for one sample).
I worked with my notebook, an its weak AC Audio hardware. My desktop computer is with all my CDs : away. Headphone is a Philips SBC-HP910.
• The most shocking results are Nero AAC poor performances. The quality is terribly poor. Faac isn’t for the moment a good encoder ; Nero, on my 18 samples, isn’t really better, but is slower and with a slightest higher average weight… I suspect (and hope) that a bug in current version is responsible. “Dorilla” sample should be an interesting one; “cello” too, but in a more common way. “Passacaglia” sample was never annoying, except with Nero encoder: maybe a problem with background noise, noticed last week with the “biniou” sample.
Interesting thing to note: PsyTel is better for my ears than Nero. Only one sample encoded with Nero ancestor was considered worse than with latest Nero encoder (“hornpipe” sample).
ADDED ONE WEEK LATER: The biggest and unexpected flaws I heard, and badly rated, with Nero AAC, are removed with the latest stable and official AAC encoder, 184.108.40.206. See
here for complete report and additional test]. Nevertheless, the more usual distortions are still present.
• Lame mp3 isn’t a bad choice for this music, and for my ears. Better than Faac, Nero and PsyTEL AAC, not too far from overall MPC and Vorbis performances.
• Vorbis surprised me : on Roberto’s multiformat 128 test, vorbis wasn’t too competitive, due to hiss/HF boost and dirty feeling it gives to me. Here, problem wasn’t always perceptible. Nevertheless, on some samples, this added noise gives the impressions of coarse manufactured instruments, rough tones, etc… Good ranking for vorbis ; excessive bitrate consumption will be partially corrected with a conventional setting (-b 4). Few pre-echo problems : I used GT3b1 encoder (but it was maybe not the best choice – I forgot that foobar2000 component was based on this particular –and particularly good– library)
• MPC was very good during Roberto’s test; with classical (baroque) only, performances are not so exciting. Nevertheless, mpc stays competitive in quality area. Unfortunately, and as expected, average bitrate isn’t competitive at all: strings, harpsichord, and ‘tonal music’ in general are bitrate greedy.
• AAC QuickTime is very pleasant. I must confess that I’m a bit disappointed: I hoped better results. Some distortions (flanging) are still noticeable with this very good encoder. Best encoder on 6 samples (33%).
With classical, QuickTime is in my opinion the very best AAC encoder at mid-bitrate.
• WMA9 PRO : I expected very good results. They are excellent. Very few flaws (more annoying was the unstable noise with “Requiem” ultra-quiet part). I couldn’t distinguish the encoded from the reference on 4 samples. Best format for 10 samples (55%).
This new encoder fits to my ears :-)
Note : I’ve got a very limited internet connection. I will try to upload the 18 samples, but I need time for it. I will begin this night (i.e. in 6 hours), with the most important one (useful, in my opinion, for Ivan). Then, I will try to finish the next week on a DSL connection. Samples will be hosted on HA server (Upload Forum). Thanks for comprehension.
Note.2 : bitrate table will follow. Total size of all files are at the bottom of the previous table.
Note.3 : I posted this result in AAC forum, just because 4 different AAC encoders were tested, though it wasn't the goal of the test.
EDIT : total size for PsyTEL and QuickTime are changed (html table only) ; I forgot to include one sample.
This post has been edited by guruboolez: Oct 18 2003, 11:48
Oct 11 2003, 17:02
Joined: 1-July 03
Member No.: 7495
Fascinating results. Especially interesting to me since you used the same Vorbis -q setting I use for normal listening (though classical is in the minority of my listening, and I use Post 1.0 CVS rather than GT3b1, so my nominal bitrate's a little lower). Psy-models are different between the two versions, so I can't extrapolate too much to how Post 1.0 CVS could compare in this test, but these results are still important to know.
I'm very surprised to see the WMA-Pro results. Not that you rated it the highest overall, but how dramatic of a difference you found between it and most of the other formats (only 0.28 between it and second place QT AAC, but 0.71 between it and tied-for-third Vorbis and MPC). If/when WMA-Pro gets decent hardware support, it'll be a serious contender with this kind of performance (if it's anything close in other genre's as well, that is).
Thanks for the work you did on this, guruboolez!
This post has been edited by ScorLibran: Oct 11 2003, 17:04
|Lo-Fi Version||Time is now: 29th August 2015 - 13:49|