IPB

Welcome Guest ( Log In | Register )

Personal multiformat listening test at ~130 kbps, based on classical (baroque) music only
guruboolez
post Oct 11 2003, 16:31
Post #1





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



INTRODUCTION

During a sleepless night, I decided to perform my own listening test, with samples based on my musical tastes. I chose a friendly bitrate encoding (~130 kbps) : in my opinion, it’s a good compromise for limited hardware space (notebook) and good quality (my preliminary experience with lower bitrate was catastrophic). Of course, testing encoding at this bitrate range is much easier. I also had in mind the idea to build a comfortable library for my notebook, and the idea to clarify some feelings I had about different encoders, on my music only. Among them :
- I suspected lame mp3 to be a good audio format on some (quiet) situation, and to not suffering as much as with pop, metal or electronic music.
- I suspected Nero AAC to be simply worse than Lame MP3 ; I did some encodings on common music the last times, and I was disappointed by the poor results I obtained for a VBR setting (-streaming). I couldn’t conclude on Nero worse quality based on three or four samples. This extend test will be more conclusive.
- I suspected QuickTime AAC and WMA 9 PRO to reach a near-transparency quality at this bitrate.



SAMPLES

Unfortunately, I worked with a limited choice of samples. With the start of the new university year, I had to leave my summer house. My discs are not for the moment in my new apartment : only the few one I bought since September. I choose samples from them, and completed the gallery with samples I burned on CD-R some times ago. Most of them are music from baroque period. It’s my favorite one. Therefore, I couldn’t materially built a coherent sample library for this test with romantic or contemporary music. That’s why there’s no piano, no Beethoven, no electronic samples here. That’s why, too, so many harpsichord (hard-to-encode instrument) is present, during singing, sacred, chamber or orchestral music. I did my best to find some samples without this instrument : more of the half.
Therefore, this test only had a limited impact: results are useful for baroque music only ; they may be different for other kind of “classical” ; and they must be totally different for jazz, country, R&B or anything else. Of course, notation is mine, dependant on my own subjectivity, my on hardware setting, and on the mood I had during the test.



ENCODERS


I encoded them with foobar2000 (except for WMA9 and PsyTEL AAC). All encodings were replaygained, original too. Decoding stage include the calculated gain (without dithering – maybe not the best choice). I didn’t make any offset correction (needed for some AAC encoding).
Challengers are :
• AAC Faac (don’t know the exact version : I used Case diskwriter component bundled with foobar2000 0.7.1 beta 4), quantizer 128, LC, m/s coding and with TNS.
• AAC Nero (aac.dll 2.5.6.3 & aacenc32.dll 2.5.5.8) : -streaming, LC, High quality
• AAC PsyTEL 2.15 –streaming [ADDED ONE WEEK LATER: I did a big mistake, and placed the good aacenc32.dll in the wrong location Therefore, I tested an old encoder : 2.5.1.6. Some odd artifacts (Dorilla, Passacaglia) are not present in 2.5.5.8. See here for complete report and additional test]
• AAC QuickTime 6.3, CBR 128, High Quality
All AAC files were decoded with faad2 (foobar2000)
• MP3 Lame 3.90.3 –alt-preset 134
• MPC 1.14 –quality 4 –xlevel
• OGG vorbis GT3b1 –q 4.25
• WMA 9 PRO VBR-2pass 128 kbps

(I chose abr 134 in order to be close to 128 kbps, because ABR mode isn’t reliable with most classical stuff. I used –b 4.25 for Vorbis, according to the average size collected some times ago, before Roberto’s test: it was a small mistake).


DECODING & PLAYBACK


All files (including reference) were replaygained with foobar2000, and then decoded with calculated gain and without dithering, No offset correction was applied (needed for correcting some AAC encoding offset) : I didn’t have the correct value.
Note than one sample (« Laudate pueri ») wasn’t properly encode/decode with PsyTEL : 15 seconds are missing, due to a bug (« negative scalfactor… »). I rated the quality for the first part only.

Comparisons were made with ff123 tool (ABC/HR). I didn’t take the time for ABXing (except for one sample).

I worked with my notebook, an its weak AC Audio hardware. My desktop computer is with all my CDs : away. Headphone is a Philips SBC-HP910.



RESULTS





(few) COMMENTS


• The most shocking results are Nero AAC poor performances. The quality is terribly poor. Faac isn’t for the moment a good encoder ; Nero, on my 18 samples, isn’t really better, but is slower and with a slightest higher average weight… I suspect (and hope) that a bug in current version is responsible. “Dorilla” sample should be an interesting one; “cello” too, but in a more common way. “Passacaglia” sample was never annoying, except with Nero encoder: maybe a problem with background noise, noticed last week with the “biniou” sample.
Interesting thing to note: PsyTel is better for my ears than Nero. Only one sample encoded with Nero ancestor was considered worse than with latest Nero encoder (“hornpipe” sample).
ADDED ONE WEEK LATER: The biggest and unexpected flaws I heard, and badly rated, with Nero AAC, are removed with the latest stable and official AAC encoder, 2.5.5.8. See
here for complete report and additional test]. Nevertheless, the more usual distortions are still present
.


• Lame mp3 isn’t a bad choice for this music, and for my ears. Better than Faac, Nero and PsyTEL AAC, not too far from overall MPC and Vorbis performances.

• Vorbis surprised me : on Roberto’s multiformat 128 test, vorbis wasn’t too competitive, due to hiss/HF boost and dirty feeling it gives to me. Here, problem wasn’t always perceptible. Nevertheless, on some samples, this added noise gives the impressions of coarse manufactured instruments, rough tones, etc… Good ranking for vorbis ; excessive bitrate consumption will be partially corrected with a conventional setting (-b 4). Few pre-echo problems : I used GT3b1 encoder (but it was maybe not the best choice – I forgot that foobar2000 component was based on this particular –and particularly good– library)

• MPC was very good during Roberto’s test; with classical (baroque) only, performances are not so exciting. Nevertheless, mpc stays competitive in quality area. Unfortunately, and as expected, average bitrate isn’t competitive at all: strings, harpsichord, and ‘tonal music’ in general are bitrate greedy.

• AAC QuickTime is very pleasant. I must confess that I’m a bit disappointed: I hoped better results. Some distortions (flanging) are still noticeable with this very good encoder. Best encoder on 6 samples (33%).
With classical, QuickTime is in my opinion the very best AAC encoder at mid-bitrate.

• WMA9 PRO : I expected very good results. They are excellent. Very few flaws (more annoying was the unstable noise with “Requiem” ultra-quiet part). I couldn’t distinguish the encoded from the reference on 4 samples. Best format for 10 samples (55%).
This new encoder fits to my ears :-)




Note : I’ve got a very limited internet connection. I will try to upload the 18 samples, but I need time for it. I will begin this night (i.e. in 6 hours), with the most important one (useful, in my opinion, for Ivan). Then, I will try to finish the next week on a DSL connection. Samples will be hosted on HA server (Upload Forum). Thanks for comprehension.

Note.2 : bitrate table will follow. Total size of all files are at the bottom of the previous table.

Note.3 : I posted this result in AAC forum, just because 4 different AAC encoders were tested, though it wasn't the goal of the test.

http://membres.lycos.fr/guruboolez/AUDIO/t...early/early.htm

EDIT : total size for PsyTEL and QuickTime are changed (html table only) ; I forgot to include one sample.

This post has been edited by guruboolez: Oct 18 2003, 11:48
Go to the top of the page
+Quote Post
 
Start new topic
Replies
Ivan Dimkovic
post Oct 11 2003, 21:58
Post #2


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



QUOTE
One made fun of me because I used the WMA with the DRM...and now one finally proves the superiority of the WMA


Well, on some other music genres WMA is not that superior smile.gif But for classical music it surely is not bad smile.gif
Go to the top of the page
+Quote Post

Posts in this topic
- guruboolez   Personal multiformat listening test at ~130 kbps   Oct 11 2003, 16:31
- - Ivan Dimkovic   Nero's performance on tonal instruments is due...   Oct 11 2003, 16:43
- - mdmuir   Once again Guru, you grace us with your amazing ea...   Oct 11 2003, 16:47
- - Ivan Dimkovic   QUOTE I also find it interesting you find the Nero...   Oct 11 2003, 16:53
- - ScorLibran   Fascinating results. Especially interesting to me...   Oct 11 2003, 17:02
- - JohnV   QUOTE I also find it interesting you find the Nero...   Oct 11 2003, 18:05
- - JohnV   Another thing. With this kind of music CBR/ABR mod...   Oct 11 2003, 19:49
- - mdmuir   Maybe by the time I am ready to upgrade to Nero 6....   Oct 11 2003, 20:46
- - guruboolez   QUOTE It would have been interesting to see how wm...   Oct 11 2003, 21:16
- - Nick Jr III   I have my revenge...thanks to guruboolez. Thank yo...   Oct 11 2003, 21:46
- - Ivan Dimkovic   QUOTE One made fun of me because I used the WMA wi...   Oct 11 2003, 21:58
- - guruboolez   Nick Jr III> Don't forget that WMA PRO is n...   Oct 11 2003, 22:04
- - Nick Jr III   QUOTE (Ivan Dimkovic @ Oct 11 2003, 03:58 PM)...   Oct 11 2003, 22:15
- - JohnV   QUOTE God heard me: victory!  WMA Rules...   Oct 11 2003, 22:21
- - Nick Jr III   QUOTE (guruboolez @ Oct 11 2003, 04:04 PM)Try...   Oct 11 2003, 22:26
- - Ivan Dimkovic   QUOTE I would like to know for which type of music...   Oct 12 2003, 00:02
- - Nick Jr III   QUOTE (Ivan Dimkovic @ Oct 11 2003, 06:02 PM)...   Oct 12 2003, 09:26
- - Ivan Dimkovic   @128 - http://audio.ciara.us/test/128extension/res...   Oct 12 2003, 10:18
- - guruboolez   During the 64 kbps listening test of Roberto, WMA9...   Oct 12 2003, 11:07
- - Nick Jr III   QUOTE (Ivan Dimkovic @ Oct 12 2003, 04:18 AM)...   Oct 12 2003, 14:07
- - Joe Bloggs   In the meantime, Microsoft is still teh evil(tm).   Oct 12 2003, 14:38
- - Garf   QUOTE (JohnV @ Oct 11 2003, 07:05 PM)QUOTE I ...   Oct 12 2003, 16:00
- - guruboolez   I finally had time enough for a specific WMA compa...   Oct 13 2003, 12:45
- - JohnV   QUOTE (guruboolez @ Oct 13 2003, 02:45 PM)I f...   Oct 13 2003, 13:26
- - guruboolez   Yes. WMA9 PRO VBR Q90 is bitrate friendly with cla...   Oct 13 2003, 13:42
- - JohnV   QUOTE (guruboolez @ Oct 13 2003, 03:42 PM)I k...   Oct 13 2003, 13:53
- - guruboolez   QUOTE (JohnV @ Oct 13 2003, 01:53 PM)QUOTE (g...   Oct 13 2003, 14:05
- - JohnV   QUOTE (guruboolez @ Oct 13 2003, 04:05 PM)Yes...   Oct 13 2003, 14:27
- - spoon   It matters not how it performs quality wise on oth...   Oct 13 2003, 15:04
- - ScorLibran   IMHO, I think it actually *is* a real world scenar...   Oct 13 2003, 15:45
- - rjamorim   I also agree with guruboolez' methodology. He...   Oct 13 2003, 15:59
- - guruboolez   CODEBased on your file sizes, the Nero release cod...   Oct 13 2003, 16:14
- - JohnV   QUOTE (rjamorim @ Oct 13 2003, 05:59 PM)Also,...   Oct 13 2003, 17:27
- - rjamorim   QUOTE (JohnV @ Oct 13 2003, 01:27 PM)I don...   Oct 13 2003, 20:36
- - JohnV   QUOTE (rjamorim @ Oct 13 2003, 10:36 PM)MPC r...   Oct 14 2003, 05:58
- - askoff   I think JohnV is at least 99% right. This same thi...   Oct 14 2003, 20:05
- - Soren   While we are in WMA topic, how can we encode wma p...   Oct 14 2003, 22:47
- - bidz   QUOTE (Soren @ Oct 14 2003, 01:47 PM)While we...   Oct 15 2003, 03:28
- - guruboolez   I see two different kind of controversy here. • F...   Oct 17 2003, 22:44
- - guruboolez   The new .dll of Ivan Dimkovic I tested this week g...   Oct 17 2003, 22:44
- - JohnV   QUOTE (guruboolez @ Oct 18 2003, 12:44 AM)The...   Oct 17 2003, 22:58
- - guruboolez   QUOTE (JohnV @ Oct 17 2003, 10:58 PM)I don...   Oct 17 2003, 23:06
- - JohnV   QUOTE (guruboolez @ Oct 18 2003, 01:06 AM)QUO...   Oct 17 2003, 23:13
- - guruboolez   QUOTE Well, it's fine by me, as long as people...   Oct 17 2003, 23:21
- - JohnV   Thanks Guru. What comes to your explanation about...   Oct 18 2003, 03:12
- - guruboolez   Thank for your positive answer. I spent long hours...   Oct 18 2003, 11:56
- - LagunaSol   Quicktime 6.4 (released with iTunes for Windows) c...   Oct 19 2003, 00:06
- - bidz   Kind of off-topic, but still a bit on-topic (as i ...   Oct 19 2003, 03:46
- - Yodule   QUOTE (Soren @ Oct 14 2003, 05:47 PM)While we...   Oct 19 2003, 05:51
- - bidz   QUOTE (Yodule @ Oct 18 2003, 08:51 PM)With th...   Oct 19 2003, 06:22
- - tigre   QUOTE (bidz @ Oct 18 2003, 06:46 PM)To be hon...   Oct 19 2003, 09:51


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 16th September 2014 - 07:39