IPB

Welcome Guest ( Log In | Register )

2 Pages V   1 2 >  
Reply to this topicStart new topic
iPod compatible formats listening evaluation, unfinished... personal... classical music only
guruboolez
post Nov 2 2007, 20:55
Post #1





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



IMPORTANT WARNINGS

• this test is unfinished
• this test is about “classical music” samples, nothing else
• choosen VBR settings are pertinent and fair with a given set of samples or musical genre (in fact, classical music) but are unlikely to work with different musical genres
• one person and only one contributed to this test. For collective listening tests, go on Roberto Amorim and Sebastian Mares websites


INTRODUCTION

My latest fad is to ask Santa Clauss one of those big iPod (160 GB) and then fit my whole musical collection and access to every CD of mine everywhere and without any computer help. My plan is to have everything at ~130 kbps which should be enough in most situation.
My last evaluation at this bitrate is two years old now. There wasn't any revolution in the meantime but all tested encoders are now outdated. That's why I decided to evaluate current encoders a second time and to restrict the test to iPod compatible formats: MP3 and AAC. Excellent formats such as Vorbis or WMAPro are for this sole reason NOT included in this 2007 evaluation.

(Un)fortunately new encoders were released or announced before the test completion (which needs almost 2 full weeks to be performed): a new LAME beta version, a new QuickTime AAC on Macintosh system. I wouldn't spend so many hours to test the 80 remaining samples with encoders that are not up-to-date anymore. I therefore cancelled the test. Some results are nevertheless very interesting - that's why I decided to post them on the forum.

TEST SETTINGS

As previous tests of mine, 150 classical music samples are supposed to feed my curiosity. I tested the half.
Hardware settings are: Terratec DMX6Fire 24/96 soundcard, Onkyo R-A5 Amplifier, Beyerdynamic DT-531 headphones. Testing rythm: ~10 samples per day.

CHOICE OF ENCODERS AND POOLS

I decided to not close the door to any modern encoders. That's why Fraunhofer MP3 iTunes MP3, Helix [Real] MP3, Coding Technologies [Winamp] AAC encoders are competing here. But all couldn't be seriously compared each others and accurately ranked. Not 150 times at least. A pre-selection was needed and that's why I started the test with smaller MP3 and AAC pools (25 samples in each).

---- In MP3 pool:
Fraunhofer v.1.4 (encoder date: 2007.05.18 ; package from 2007.07.11): VBR -m3 -q1
Helix MP3 v.5.1: VBR -V65
iTunes v.7.4.3.1 MP3: VBR 128 highest
LAME v.3.98 beta 5: VBR -V5



No real surprises here. LAME is better than any other competitors with a confidence >95% and will therefore compete with AAC in final test. iTunes and Fhg performances are similar and aren't globally bad; but both are more unstable (or less robust against encoding difficulties) than LAME. Helix is considerably worse and finishes last.


---- In AAC pool

I exempted iTunes from the selections due to its excellent past performances. It will be opposed in the global evaluation with the winner of this pool. I know two serious challengers: Coding Technologies and Nero Digital. The latter is represented in the pool by two releases: the last one and the previous one. I'm very suspicious about the last Nero's encoder since I discovered problems with it. This test and the direct comparison with the previous Nero Digital release is a good occasion to confirm or infirm my suspicions.

Coding Technologies[encoder built in Winamp 5.5 beta, bit-identical output to final version]: CBR 128
Nero Digital v.1.0.7.0 (February 2007): VBR -q 0,45
Nero Digital v.1.1.3.4 (Aug 2007): VBR -q 0,44 (higher bitrate and better performance than -q0.45…)



This test confirms the existence of a quality gap between Nero Digital 1.1.34.2 and the previous version. This gap is really huge for my taste (and with classical music I recall). Something bad happens with last version and I hope it will fixed. Coding Technologies AAC's performances are rather good, especially for a CBR encoder. Nero 1.0.7.0 wins and will be opposed to iTunes AAC and LAME.

The final test will therefore be: iTunes AAC + Nero Digital 1.0.7.0 + LAME 3.98b5 + 2 anchors



CHOICE OF ANCHORS

For once I decided to be more practical than “scientifical” in the choice of anchors and take some risks.

As low anchor I used 96 kbps AAC. It's a (small) risk because this setting might give similar results to MP3 at 130 kbps. It's unlikely but not impossible. I was also very curious to check how would perform ~100 kbps by itself, and when compared to MP3. If low anchor appear to be “good enough” why not using it to fill a digital jukebox? iTunes can't be used here because last version forces 32000 Hz sampling rate (too easily noticeable for my taste on most situation). Nero Digital is configurable and offers VBR 96 kbps encodings. For obvious reasons I avoided last Nero implementation.

As high anchor I used 160 kbps MP3 (LAME -V4). The risk is higher IMO and I wouldn't be really surprised if any AAC implementation would reach or surpass even with a penalty of 30 kbps. But if MP3@160 would outperfom AAC@130 I would seriously consider the most compatible and universal format as final encoding choice for my future chritmas present.
• Nero Digital 1.0.7.0 -lc -q0,25
• LAME 3.95b5 -V4


The following evaluation can be read as:
- Nero AAC vs iTunes AAC vs LAME MP3
- Nero AAC at 100 kbps vs Nero AAC at 130 kbps
- LAME -V5 vs LAME -V4
- 100 kbps AAC vs 130 kbps MP3
- 130 kbps AAC vs 160 kbps MP3




RESULTS

Note: only plots are posted here. As unfinished test I won't spend too much time to format all results individually as I did in previous tests (like here and here).


I. CLASSICAL: 5 electronic/artificial samples



5 samples aren't enough to get statistically pertinent conclusions. With confidence we can say that LAME at 160 kbps is better than Nero AAC at 100 kbps (what a surprise...). I can also add that none of current AAC implementation is able to give good anything better than average results on these critical samples. I regret aoTuV Vorbis performance which was able to match (and even surpass) LAME -V2 at 128 kbps (nominal) (see here to get 2 years old plot).


IIa. CLASSICAL: 30 orchestral & chamber samples on modern instruments



With this group of 30 samples we have what people would consider as “classical” music: chamber and orchestral various compositions on modern instruments.
And here the first big change since my last evaluation: while iTunes still performs well (average mark is in progress) it's now clearly and statistically inferior to Nero Digital. The latter offers similar results to LAME at 160 kbps and is the only AAC implementation that appear to be more efficient than MP3. I must also recall that Nero Digital -q0,45 doesn't necessary offers ~130 kbps with other musical contents.


IIa. CLASSICAL: 30 orchestral & chamber samples on period instruments



Early instruments are usually harder to encode properly. It's confirmed here. LAME performances are significantly worse here. It's strange: 2 years ago LAME 3.98 alpha 2 was more constant and offered similar results between modern and period instruments recording. At least with -V5... With higher settings LAME also showed stronger regression in the past (-V2 with alpha 2 and now V4 with beta 5). LAME is here statistically tied with low anchor.
iTunes AAC is very constant. Nero Digital is now statistically tied with iTunes and with high anchor but ends this group with the best mark. It confirms on 30 additionnal samples how great it is.



VERY SMALL CONCLUSION


I haven't tested the whole “vocal music” group (30 samples) and most samples of the “solo instruments” (60). But intermediate results (20 samples on 60) of the later group are confirming previous ones:

LOW ANCHOR: 3.1
iTUNES AAC: 4.1
LAME MP3: 3.2
Nero AAC: 4.4
HIGH ANCHOR: 4.0

In other words, Nero Digital 1.0.7.0 -q0.45 looks like a jewel: better than iTunes 7.4.3.1 at the same bitrate, as good as MP3 at 160 kbps. Now it would be more than interesting to compare it to the newest QuickTime version (only available on Mac OS 10.5 ATM) and also expecting from Nero Team to fix current issues on 1.1.34.2.
… and hope from Apple they will correct all iPod classic (good name) issues with a new firmware.



___
appendix

bitrate table (150 full tracks, 16 hours of music from 150 classical music CD):
http://audiotests.free.fr/tests/2007.11/iPodTEST_bitrate.ods

2 gif files illustrating worrying ringing artefacts (+ agressive lowpass/ATH value) with Nero 1.1.34.2:
E22 sample (0-22000 Hz)
V19 sample (10000-19000 Hz zoom)
Go to the top of the page
+Quote Post
/mnt
post Nov 2 2007, 21:39
Post #2





Group: Members
Posts: 697
Joined: 22-April 06
Member No.: 29877



A great listening test. This shows that iTunes mp3 encoder is not that bad and Helix carries the Xing imfamus legacy biggrin.gif. I have found that the latest verison of Nero to be poor compared to the Feb 2007 version a few months ago with some AAD mixed samples such Iron Man by Black Sabbath which produces a annoying metallic noise in the background.

This post has been edited by /mnt: Nov 2 2007, 21:39


--------------------
"I never thought I'd see this much candy in one mission!"
Go to the top of the page
+Quote Post
guruboolez
post Nov 2 2007, 22:04
Post #3





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (/mnt @ Nov 2 2007, 22:39) *
A great listening test. This shows that iTunes mp3 encoder is not that bad and Helix carries the Xing imfamus legacy

Thanks.
I guess it would be better to wait for Sebastian's MP3 test (more listeners, different samples, but probably lower VBR setting) before stigmatising Helix.
iTunes also surprised me. It reacted differently from Fhg's encoder but finally ended with the same overall quality. It's worth noting that I used -m3 as VBR setting for Fraunhofer (which seems to be closer to 160 kbps than 130 kbps with most popular musical material - cf. AlexB bitrate table on the thread dedicated to the upcoming listening test). Would -m4 be as good as iTunes encoding? Maybe iTunes MP3 will appear to be the second best MP3 encoder in Sebastian's test.

This post has been edited by guruboolez: Nov 2 2007, 22:05
Go to the top of the page
+Quote Post
/mnt
post Nov 2 2007, 22:16
Post #4





Group: Members
Posts: 697
Joined: 22-April 06
Member No.: 29877



QUOTE (guruboolez @ Nov 2 2007, 22:04) *
QUOTE (/mnt @ Nov 2 2007, 22:39) *

A great listening test. This shows that iTunes mp3 encoder is not that bad and Helix carries the Xing imfamus legacy

Thanks.
I guess it would be better to wait for Sebastian's MP3 test (more listeners, different samples, but probably lower VBR setting) before stigmatising Helix.
iTunes also surprised me. It reacted differently from Fhg's encoder but finally ended with the same overall quality. It's worth noting that I used -m3 as VBR setting for Fraunhofer (which seems to be closer to 160 kbps than 130 kbps with most popular musical material - cf. AlexB bitrate table on the thread dedicated to the upcoming listening test). Would -m4 be as good as iTunes encoding? Maybe iTunes MP3 will appear to be the second best MP3 encoder in Sebastian's test.

It sure is better to wait for Sebastian's since some encoders are likely to perform better at different genres and samples, but am suprised that iTunes mp3 performed well since Apple would spend more time devloping their AAC encoder.


--------------------
"I never thought I'd see this much candy in one mission!"
Go to the top of the page
+Quote Post
Gow
post Nov 2 2007, 22:30
Post #5





Group: Members
Posts: 233
Joined: 14-June 06
From: Fort Wayne, IN
Member No.: 31824



Made me re-think the use of Nero AAC 1.1+ instead of using 1.0.7.0. Think I will go back to using the older Feb 2007 release over Aug 2007 release. Good test by the way, well done.

EDIT: Its use for Classical Music that is.

This post has been edited by Gow: Nov 2 2007, 22:39


--------------------
Zune 80, Tak -p4 audio library, Lossless=Choice
Go to the top of the page
+Quote Post
guruboolez
post Nov 2 2007, 23:40
Post #6





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



I've merged all 85 scores and analysed it with friedman:
CODE

FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Tukey HSD analysis

Number of listeners: 85
Critical significance: 0.05
Tukey's HSD: 0.226

Means:

Nero 160kbps iTunes LAME 100kbps
4.33 4.24 4.00 3.47 3.17

-------------------------- Difference Matrix --------------------------

160kbps iTunes LAME 100kbps
Nero 0.089 0.331* 0.856* 1.156*
160kbps 0.241* 0.767* 1.067*
iTunes 0.526* 0.826*
LAME 0.300*
-----------------------------------------------------------------------

Nero is better than iTunes, LAME, 100kbps
160kbps is better than iTunes, LAME, 100kbps
iTunes is better than LAME, 100kbps
LAME is better than 100kbps



Graphical results looks like this:




It is safe (for me) to say that:
• Nero Digital (Feb 2007) is better on a wide selection of classical music than iTunes AAC and LAME MP3
• AAC@130 kbps (Nero only, classical music) may offer similar (and even slightly better) quality than top-MP3@160 kbps
• iTunes AAC is better than LAME -V5
• AAC@100 kbps is clearly lower quality than top-MP3@130 kbps.
• Nero@130 kbps > Nero@100 kbps
• MP3@160 > MP3@130 (trivial conclusion but would it appear on a collective listening tests involving several “non-critical” listeners?)

=> Nero Digital AAC (Feb 2007) is what I'll use first, though it's not perfect at this quality setting (-q0,45).

This post has been edited by guruboolez: Nov 2 2007, 23:41
Go to the top of the page
+Quote Post
twostar
post Nov 3 2007, 03:36
Post #7





Group: Members
Posts: 487
Joined: 5-August 02
From: Manila
Member No.: 2939



great test guru! wish i hadn't deleted my copy of Nero Digital v.1.0.7.0

This post has been edited by twostar: Nov 3 2007, 04:41
Go to the top of the page
+Quote Post
clintb
post Nov 3 2007, 09:09
Post #8





Group: Members
Posts: 182
Joined: 4-July 02
From: Sunnyvale, CA
Member No.: 2472



QUOTE (twostar @ Nov 2 2007, 21:36) *
great test guru! wish i hadn't deleted my copy of Nero Digital v.1.0.7.0

Here ya go!

http://www.brothersoft.com/mp3_audio/rippe...udio_59677.html
Go to the top of the page
+Quote Post
Enig123
post Nov 3 2007, 13:39
Post #9





Group: Members
Posts: 208
Joined: 11-April 02
Member No.: 1749



It's more fun to get the upcoming new Apple's vbr aac encoder in this test in the near future. With all fixes of the regressions found maybe the next version of Nero digital would get much better, which is also worth waiting.
Go to the top of the page
+Quote Post
richard123
post Nov 3 2007, 15:14
Post #10





Group: Members
Posts: 348
Joined: 9-January 03
Member No.: 4498



Using Nero Digital v.1.0.7.0 (February 2007): -q 0.45 I get bit rates around 155k, noticeably higher than 130k. Am I getting unusual results?

This post has been edited by richard123: Nov 3 2007, 15:14
Go to the top of the page
+Quote Post
guruboolez
post Nov 3 2007, 15:19
Post #11





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



I don't think I have enough material to build a valid bitrate for non-classical music; from what I've tested (some old heavy metal stuff) it's indeed closer to 160 kbps than 130 kbps.
You finding are probably more a rule than an exception. I mentioned it in the test: the choosen VBR settings are based on average bitrate obtained with classical music, which is often (but not always) bitrate-friendly compared to modern or popular (i.e louder, less dynamical) music.

This post has been edited by guruboolez: Nov 3 2007, 15:20
Go to the top of the page
+Quote Post
richard123
post Nov 3 2007, 16:05
Post #12





Group: Members
Posts: 348
Joined: 9-January 03
Member No.: 4498



A small sample of classical tracks was generating values around 145k, but more testing brought the average closer to your range. Rock tends to be higher.

Great testing
Go to the top of the page
+Quote Post
/mnt
post Nov 3 2007, 16:17
Post #13





Group: Members
Posts: 697
Joined: 22-April 06
Member No.: 29877



Its a shame that Nero Digital v.1.0.7.0 has a track length bug that effects playes such as Quicktime, GStreamer FAAD and XMMS FAAD plugins which causes the player to cut off at the last few secends. Which has been fixed on Aug 2007 and also listed on their release notes.

This post has been edited by /mnt: Nov 3 2007, 16:20


--------------------
"I never thought I'd see this much candy in one mission!"
Go to the top of the page
+Quote Post
Squeller
post Nov 5 2007, 10:34
Post #14





Group: Members
Posts: 2351
Joined: 28-August 02
Member No.: 3218



I stay with lossless and wait for an update because in some very silent parts (also classical here) the encoder chooses very low bitrates even at q6, resulting in kind of digital hiss artifacts.
Go to the top of the page
+Quote Post
guruboolez
post Nov 5 2007, 12:43
Post #15





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



I just noticed it, in real listening conditions and not during a listening test. It occurs with quiet piano recordings with -q 0,45. Bitrate is low (I expect this from a VBR encoder), between 80 and 100 kbps; but quality is sometimes low too (I don't expect this from VBR which is supposed to maintain a constant quality) sad.gif
I reached the best size/quality ratio with LAME -V5: ~100 kbps with this material and close to transparency files (but not always transparent). The most reliable VBR LC-AAC encoder in this situation seems to be iTunes which isn't (unfortunately) the most efficient one.
I'm now mixing the encoders for my growing M4A library.
Go to the top of the page
+Quote Post
dbAmp
post Nov 9 2007, 01:15
Post #16





Group: Members
Posts: 107
Joined: 10-October 05
Member No.: 25035



Guruboolez, have you tried the new AAC encoder in iTunes 7.5 / QuickTime 7.3 yet? I've encoded about 20 CDs from various genres to do comparisons... and I'm finding that at the 128 VBR setting, I'm getting lower bitrate and lower quality files than I did with the same settings in iTunes 7.4 / QuickTime 7.2. I'm not entirely sure what the changes are... but this new encoder seems to be a step backward... especially when you consider that upping the settings to 160 VBR creates a significantly higher bitrate and larger sized file.
Go to the top of the page
+Quote Post
guruboolez
post Nov 9 2007, 02:01
Post #17





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



No, I haven't try yet and I don't think I can test it before christmas.
But as I plan to use iTunes soon for encoding several discs (piano music for which Nero Digital lowers the bitrate as well as the sound quality; harpsichord discs too for which iTunes@160 sounds better than Nero@~170) it would probably wise for me to check for possible issues with the newer encoder, especially after your interesting report.

This post has been edited by guruboolez: Nov 9 2007, 02:01
Go to the top of the page
+Quote Post
Enig123
post Nov 9 2007, 02:35
Post #18





Group: Members
Posts: 208
Joined: 11-April 02
Member No.: 1749



There's actually four encoding mode with the latest QuickTime aac codec: true vbr, cbr, abr, constrained vbr. I'm not sure which mode is the default vbr mode for iTunes.

guruboolez, I suggest you checking all 3 vbr modes when you do the test.
Go to the top of the page
+Quote Post
guruboolez
post Nov 9 2007, 02:49
Post #19





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



I will only use and therefore test what I can use in batch encoding. An encoding mode available in QuickTime only is pointless for CD ripping / batch encoding purpose - unless someone will provide a tool alla iTunes_encode to exploit it in third-party apps.
I would be disappointed if iTunes didn't include the most interesting (true VBR) encoding mode available in QuickTime.

This post has been edited by guruboolez: Nov 9 2007, 02:50
Go to the top of the page
+Quote Post
dbAmp
post Nov 9 2007, 04:39
Post #20





Group: Members
Posts: 107
Joined: 10-October 05
Member No.: 25035



QUOTE (Enig123 @ Nov 8 2007, 21:35) *
There's actually four encoding mode with the latest QuickTime aac codec: true vbr, cbr, abr, constrained vbr. I'm not sure which mode is the default vbr mode for iTunes.

guruboolez, I suggest you checking all 3 vbr modes when you do the test.


QUOTE (guruboolez @ Nov 8 2007, 21:49) *
I will only use and therefore test what I can use in batch encoding. An encoding mode available in QuickTime only is pointless for CD ripping / batch encoding purpose - unless someone will provide a tool alla iTunes_encode to exploit it in third-party apps.
I would be disappointed if iTunes didn't include the most interesting (true VBR) encoding mode available in QuickTime.


I'm fairly certain that iTunes is and always has used the constrained VBR setting.
Go to the top of the page
+Quote Post
IgorC
post Nov 9 2007, 06:01
Post #21





Group: Members
Posts: 1532
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



QUOTE
Nero Digital 1.0.7.0 -lc -q0,25

It's low anchor settings. Doesn't default -q 0.34/0.35 produce 96-100 kbit/s LC?
Go to the top of the page
+Quote Post
guruboolez
post Nov 9 2007, 12:25
Post #22





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



I'm fairly certain that iTunes is and always has used the constrained VBR setting.
I share this. The new iTunes uses a constrained VBR setting too; it's confirmed here.

It's low anchor settings. Doesn't default -q 0.34/0.35 produce 96-100 kbit/s LC?
I recall that I'm not using -q,025 but -q0.25 -lc. The -lc or -he settings forces the use of a different bitrate scale.
I tried to reach ~96kbps with the regular scale (it was indeed with something near to -q0.35) and the encoder forces the use of HE-AAC - which is pointless on an iPod compatible format setting. That's why I had to find the adequate -q level with the forced -lc mode.
Go to the top of the page
+Quote Post
Polar
post Dec 14 2007, 09:57
Post #23





Group: Members
Posts: 266
Joined: 12-February 04
Member No.: 11970



Christmas is coming up, and so is that iPod of yours ;-)
Any progress on your interesting effort, Guruboolez?
Go to the top of the page
+Quote Post
zorba
post Feb 11 2008, 08:32
Post #24





Group: Members
Posts: 73
Joined: 22-December 05
Member No.: 26571



is the test finished now ?
Go to the top of the page
+Quote Post
hans-jürgen
post Mar 8 2008, 15:36
Post #25





Group: Members
Posts: 573
Joined: 2-August 02
From: Hamburg, Germany
Member No.: 2898



QUOTE (clintb @ Nov 3 2007, 09:09) *

Has only the newest version now, here's a working link for the older version:

http://download.chip.eu/de/Nero-AAC-Encode...7.0_155834.html

Thanks for your test, guru!


--------------------
myspace.com/bluezzbastardzz
myspace.com/indigorocks
Go to the top of the page
+Quote Post

2 Pages V   1 2 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 23rd July 2014 - 16:31