IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
AAC: Ahead vs Apple (end 2004), one year of progress
guruboolez
post Dec 9 2004, 08:23
Post #1





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



Last year, I’ve tested various audio formats at 130 kbps, based on a classical sample suit (15 samples), and including among different audio formats many AAC encoders. Conclusions were highly positive for QuickTime AAC which came at first place by producing the most amazing results I’ve never heard at 128 kbps. But they were also very severe for Nero AAC: quality was simply awful with most samples, and it finished on last position. Obviously, Nero AAC suffered a lot with “classical” music – and, according to the collective test leaded by rjamorim, this encoder (updated) was also inferior to Apple AAC with other kind of music. I’ve found this situation annoying, because Ahead encoder is on PC much more flexible (gapless, many GUI allowing transcoding from various source with tags report, SBR…) than Apple’s encoder. I know that Ivan Dimkovic tried to find the cause of these troubles — probably a difficult and long task. Therefore, I promised to myself to not publish any test including an unfinished AAC encoder: by respect for Ivan first, and then to let him the necessary time to tweak his encoder.

Yesterday, I’ve played a bit with the latest public encoder (2.9.9.998: final 3.0.000 is now very close). Quality is clearly far from disaster I’ve heard last year and now become more than acceptable. Nevertheless, usual problems are still audible: lifeless sound (the encoder removes some precious musical information -> hollow sounding), and sometimes severe distortions (big frequencies holes). Disappointed, I’ve decided to not investigate further, and to wait again for the next big step. For curiosity, I’ve also performed small benches (speed, bitrate), and noticed a strange thing. Though 3.0.000 should be release in the next future, there are still some incoherencies in current encoder. For example, the “fast” mode is not faster than “high” (default mode), and even stranger, bitrate could be completely different. On harpsichord tracks for example, -internet VBR profile lead to ~180 kbps with “fast” and with normal 128 kbps with “high”. Is “fast” mode a totally different codec? An experimental one? Or the future 3.0.00? Suspicious, I’ve encoded in “fast” mode some problem samples, and all quality problems noticed with the default mode completely disappeared (bitrate between “high” and “fast” were similar, and in some case really inferior: see this exemple). In order to avoid some unjustified enthusiasm, I performed quick blind tests: results were simply beyond my optimism, and they confirmed the serious quality difference existing between the two settings. Same conclusions for CBR 128: “fast” is clearly better than “high” with all samples I’ve tested. Not only better, but also excellent — and this is certainly the most interesting point.

Immediate problem with VBR “fast” is the average bitrate. Slightly higher than 128 kbps on average (tested on few tracks), it sometimes jump very high with specific tracks (not only short samples, but full albums are apparently concerned)… especially for the tracks I like the most :/. This situation is annoying in two cases:
• When filling a small flash player (the future iPod flash for example, but it could also be a problem for iPod mini), excessive bitrate will drastically reduce the amount of embedded music.
• When doing a complete listening test: people will always complain about biased results, etc…
Both problems disappear with CBR/ABR encodings. Now, if Nero CBR is close if not better – who knows? – than a VBR mode, why not compare it with current reference, iTunes AAC (CBR only), which could loose its crown. After all, there are some suspicions around Apple new encoders (since strange frequencies behaviour introduced with QuickTime 6.5.1), and Ahead encoder has a chance to win this match. A full test is definitively worth.

Therefore, I’ve decided to do this test now, and to not wait for the final release of Nero AAC 3. But I’m not interested by a basic bloody competition between Apple and Ahead products; I’d like to answer to other interesting questions, like:

• how much differences between Ahead CBR and Ahead VBR? Is VBR necessary better? On all samples? Are those few kbps added by –internet VBR mode worth compared to CBR 128?
• did iTunes progress during one year? Quality was so good last year… Or has it really worsened, as noticed or maybe feared by some people? Old iTunes winning codec had a lowpass cutoff fixed at 16 KHz whereas the new one has now reached 18000 KHz: is it really a good thing?
• how much progress for Ahead AAC? Big jump or limited improvements during one year?


In order to answer to these questions, I’ve decided to take as samples the same I used last year, and to compare directly, on the same test, the old encodings (Nero-looser-2003 & Apple-winner-2003) to the new line (Nero-2004 & Apple-2004). Therefore, challengers are:
— iTunes 4.1.052 AAC (corresponding to QuickTime 6.4) at 128 kbps [CBR]
— iTunes 4.7.042 AAC (corresponding to QuickTime 6.5.2) at 128 kbps [CBR]
— Nero AAC 2.5.9.7 at 128 kbps [CBR rather than VBR: CBR was slightly better on average than VBR on last test]
— Nero AAC 2.9.998 at CBR 128 kbps “fast”
— Nero AAC 2.9.998 at VBR –internet “fast”.

By introducing in the same arena both CBR and VBR for Ahead encoder, results could satisfy two categories of people:
— those which can’t bear bananas and orange at the same time, and refuse to consider any comparison between a CBR encoder and a VBR one. They could compare Apple and Ahead products performance at exactly the same file size.
— those which don’t see any reason to handicap complete encoders, by forcing constant bitrate when VBR is probably a more efficient (best quality and/or smaller file size) solution. They could compare Ahead at his (supposed) best encoding mode and Apple performances, which is CBR only.

Few words about methodology.

I used an Audigy2 soundcard and a Beyerdynamic DT-531 headphone. ABC/ABX tool was ff123’s ABC/HR. Following Pio2001 recommendations, I’ve ABXed all samples using fixed number of trials; but for personal reasons, I’ve choose two different values: eight when encoding is easy to betray, sixteen in other cases or when some stupid mistakes occurred on the 8 first trials. I’ve just made one exception with one sample and one encoding (Mozart – Requiem and iTunes 4.7). When I failed an ABX test, I didn’t bring the ABC notation to 5.0 (which could be considered as a logical conclusion), but I usually consider again all notations during/after ABX sessions.
Last, according to Gabriel’s recommendations I didn’t listened the first second of each sample (removed by selecting a specific range through ABC/HR). I focused the notation on the first seconds (5…10 sec.) in order to upload smaller files.


And now: results :-)



>> click on image/link for results <<


FEW COMMENTS


iTunes 4.1.052 AAC encoder was the very best encoder last year, even superior to all other tested audio formats (including vorbis or wmaPRO 2-pass). This time, the same encoder on the same samples finishes at last place! Developers were obviously very active in 2004! Old iTunes produced the worse sound on 6 samples (on 15). Last year, with less aggressive challengers, this encoder obtained 4.0 as final score (=“perceptible difference but not annoying”). Now, artefacts are less acceptable in regard of concurrence (final score: 3.0/5 = “slightly annoying”).

iTunes 4.7.042 AAC encoder was slightly inferior to his cousin on two samples only (Compostelle.wav: slight pre-echo on a bell; Mahler: additional distortions on brass), but superior on 10 other samples. In other words, the feared or claimed regression doesn’t happen, and there are even clearly audible progresses. Basing my opinion on this limited test, I don’t see any reason to keep old iTunes software. Few words about quality: sound is less lowpassed (it’s clearly audible with some specific instruments: cymbals, harpsichord…), and this amount of extra-information doesn’t increase the level of audible distortions. There’s also no ringing (at least not more than the old encoder). Some problems are unfortunately still here, like annoying pre-echo amount on a piano piece (Brahms – Hungarian Dance 6). Very nice overall quality!

Nero 2.5.9.7. Intolerable quality! It wasn’t a surprise: I used it as low anchor, and was perfect in this role (with one exception: Haendel (female), or Hercules.wav).

Nero 2.9.998 CBR. Different from Apple AAC CBR but performances are absolutely tied. Better with 7 samples, and worse with the eight others. This encoder loses a lot of point with the two harpsichord tracks (Apple’s encoder is excellent here). People could notice that this encoder is better on average (on 9 samples) than the winning encoder of last test! Amazing progress in one year!

Nero 2.9.998 VBR. Probably one of the best things I’ve never heard at this bitrate (the other strong competitor could be vorbis AoTuV: a direct comparison would be highly interesting). It outperforms both CBR encoders. Best sound on 10 samples. Never worse (which means that VBR mode is reliable and that it apparently doesn’t suffer from unusual situation, like low volume/bitrate encoding). Only problem: some bitrate inflation, sometimes strange to explain (the organ sample for example, slightly worse quality than CBR 128, but bitrate >200…300 kbps on one short moment), sometimes highly judicious (Bayle sample: perfect reproduction of this complex sound; real progress with harpsichord compared to CBR mode, but still inferior to Apple CBR). Probably some progress margin for pre-echo.


FEW STATISTICS

I fed ff123 analysis tools with the following table:
CODE
QT64    QT652    OLDNERO    NEROCBR    NEROVBR
2.0    3.0    1.0    2.0    4.5
2.5    3.0    1.5    4.0    5.0
3.0    3.5    1.5    4.0    5.0
3.0    4.0    1.0    5.0    4.5
3.0    3.2    1.0    4.0    4.0
5.0    5.0    4.4    5.0    5.0
3.5    3.0    1.0    2.5    4.5
4.0    4.5    1.5    2.0    3.5
2.5    2.5    1.5    3.0    4.5
3.0    2.5    1.5    3.5    3.5
3.0    3.0    1.5    4.5    4.0
2.5    3.3    1.5    4.0    4.0
3.0    3.8    1.5    2.0    3.0
2.5    3.5    1.6    3.0    4.5
3.2    3.7    1.4    2.7    3.2



• ANOVA conclusions

CODE
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 15
Critical significance:  0.05
Significance of data: 2.10E-014 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total               74         100.71
Testers (blocks)    14          21.40
Codecs eval'd        4          56.20   14.05   34.05  2.10E-014
Error               56          23.11    0.41
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.470

Means:

NEROVBR  QT652    NEROCBR  QT64     OLDNERO  
 4.18     3.43     3.41     3.05     1.56  

---------------------------- p-value Matrix ---------------------------

        QT652    NEROCBR  QT64     OLDNERO  
NEROVBR  0.002*   0.002*   0.000*   0.000*  
QT652             0.932    0.105    0.000*  
NEROCBR                    0.124    0.000*  
QT64                                0.000*  
-----------------------------------------------------------------------

NEROVBR is better than QT652, NEROCBR, QT64, OLDNERO
QT652 is better than OLDNERO
NEROCBR is better than OLDNERO
QT64 is better than OLDNERO


Two reliable conclusions based on my hearing and on 15 classical samples:
- old Nero AAC encoder is inferior to all other challengers
- new Nero AAC encoder, with VBR –internet and “fast” mode is superior to all other challengers.
- old iTunes, new iTunes and new Nero CBR are tied

• TUKEY PARAMETRIC conclusions

CODE
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Tukey HSD analysis

Number of listeners: 15
Critical significance:  0.05
Tukey's HSD:   0.662

Means:

NEROVBR  QT652    NEROCBR  QT64     OLDNERO  
 4.18     3.43     3.41     3.05     1.56  

-------------------------- Difference Matrix --------------------------

        QT652    NEROCBR  QT64     OLDNERO  
NEROVBR    0.747*   0.767*   1.133*   2.620*
QT652               0.020    0.387    1.873*
NEROCBR                      0.367    1.853*
QT64                                  1.487*
-----------------------------------------------------------------------

NEROVBR is better than QT652, NEROCBR, QT64, OLDNERO
QT652 is better than OLDNERO
NEROCBR is better than OLDNERO
QT64 is better than OLDNERO

Conclusions are exactly the same than ANOVA analysis.


Applauses for Apple and Ahead developers :-)


• ABX logs are available here
• SAMPLES (11 MB) are available here. Keep them: I can’t keep them very long time online.

This post has been edited by guruboolez: Dec 29 2005, 21:56
Go to the top of the page
+Quote Post
Digga
post Dec 9 2004, 08:52
Post #2





Group: Members
Posts: 1047
Joined: 28-June 03
From: on the dock of the bay
Member No.: 7423



it is always a pleasure to see you make tests guruboolez, thank you for your efforts.


--------------------
Nothing but a Heartache - Since I found my Baby ;)
Go to the top of the page
+Quote Post
guruboolez
post Dec 9 2004, 10:29
Post #3





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



EDIT: LINK for result is now working.... dry.gif I've also changed the link directing to the wrong old test.

Digga> thanks smile.gif

Ahead team> some clarifications about this “fast” encoder, which isn't faster, but seems to be a different encoder?

This post has been edited by guruboolez: Dec 9 2004, 10:35
Go to the top of the page
+Quote Post
Gabriel
post Dec 9 2004, 10:50
Post #4


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



I like those results.
Top AAC encoders seems to have made good progress. I guess that now AAC would be quite better than mp3 on a 128kbps group test.
That is also quite good for low bitrates, as previsouly (http://www.rjamorim.com/test/64test/results.html) Ahead's LC core was obviously inferior to Apple's one. In the 64k test Ahead won because of the SBR part. Let's do a few extrapolations based on the 64k test:
Look at the margin between FhG mp3 and mp3pro. This margin was only because of SBR. Now, let's assume that the LC core from Ahead is now tied to Apple's one. Add the sbr margin to the Apple rating, and you have an incredible potential rating that would score past 4!

Regarding the "fast" encoder, my guess would be that it is trying to directly compute scalefactor values instead of trying to reach an optimal combination by iterating. (but I have no knowledge if the inners of this encoder, just a guess)
Go to the top of the page
+Quote Post
SirGrey
post Dec 9 2004, 11:36
Post #5





Group: Members
Posts: 241
Joined: 8-February 04
Member No.: 11863



>>Add the sbr margin to the Apple rating, and you have an incredible potential
>>rating that would score past 4!
Heh. Would be really interesting to test.
There were a rumors, that Apple will bring us HE-AAC, but seems that it is still the rumours...
BTW, thanks, guruboolez !
Go to the top of the page
+Quote Post
QuantumKnot
post Dec 9 2004, 13:06
Post #6





Group: Developer
Posts: 1245
Joined: 16-December 02
From: Australia
Member No.: 4097



As always, it's very comprehensive and very enjoyable to read. Thanks guruboolez. smile.gif
Go to the top of the page
+Quote Post
bond
post Dec 10 2004, 00:59
Post #7





Group: Members
Posts: 881
Joined: 11-October 02
Member No.: 3523



interesting that fast produced better results than high for you guruboolez (thanks for this comparison!)

i hope ivan can give us some explanations here (i always hoped that neros pure lc-aac encoder would get much better, it was really needed)


--------------------
I know, that I know nothing (Socrates)
Go to the top of the page
+Quote Post
negritot
post Dec 10 2004, 03:46
Post #8





Group: Members
Posts: 209
Joined: 2-January 04
Member No.: 10890



As always, a very interesting read. Thanks for your work, guruboolez. Nero AAC is becoming very interesting, indeed.
Go to the top of the page
+Quote Post
DreamTactix291
post Dec 10 2004, 07:48
Post #9





Group: Members (Donating)
Posts: 552
Joined: 9-June 04
From: A place long since forgotten...
Member No.: 14572



It's nice to see Nero VBR perform so well. Thank you for this guruboolez. Because I do see AAC being more the real successor to mp3 as it has businesses backing it, it's nice to see the encoders mature as time progresses.

Oh and great work by all you programmers at Ahead. You guys really improved your codec in a fairly short amount of time.


--------------------
Nero AAC 1.5.1.0: -q0.45
Go to the top of the page
+Quote Post
Busemann
post Dec 11 2004, 19:29
Post #10





Group: Members
Posts: 730
Joined: 5-January 04
Member No.: 10970



Interesting read! Good to get confirmation on the ringing issue with QT, and big props to the nero guys smile.gif
Go to the top of the page
+Quote Post
sony666
post Dec 22 2004, 05:39
Post #11





Group: Members
Posts: 573
Joined: 22-February 02
Member No.: 1375



A highly interesting read indeed.
Thank you very much smile.gif
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 30th September 2014 - 21:59