IPB

Welcome Guest ( Log In | Register )

6 Pages V   1 2 3 > »   
Reply to this topicStart new topic
AAC at 128kbps test - FINISHED
rjamorim
post Jun 16 2003, 08:12
Post #1


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



I'd like to announce that my first listening test successfuly ended. smile.gif

The results page is available here:
http://rarewares.hydrogenaudio.org/test/aa...st/results.html
(veteran forum members will notice that page resembles to someone else's)

So far, you only get results for each sample, and overall results.

Tomorrow, I'll add fancy things like user comments, bitrates, etc.

If someone notices flaws or typos, please report in this thread or through PM. Thanks a lot.
(Also, info on the 41_30sec sample is welcome)

For those really hurried, here is the overall plot:


Once again, thanks to everyone that participated. Hopefully, there'll be more in a month. smile.gif

Kind regards;

Roberto Amorim.


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
sven_Bent
post Jun 16 2003, 08:35
Post #2





Group: Members
Posts: 691
Joined: 15-December 01
From: Denmark
Member No.: 655



wow
QT really shines wonder how it compares to psytell -streaming.


--------------------
Sven Bent - Denmark
Go to the top of the page
+Quote Post
ChrisGranger
post Jun 16 2003, 08:39
Post #3





Group: Members
Posts: 50
Joined: 31-March 03
From: Winnipeg, Canada
Member No.: 5772



Just a little typo in the last line: "Sorenson is good, but it's price is prohibitive." It should be "its" which is possessive.

Interesting test. I guess we shouldn't be too surprised to see how well the QuickTime's AAC did. I was a little surprised to see FAAC getting beaten so badly though. Would you say it's still a better alternative to 128kbps MP3? (I assume we're using this bitrate for the purpose of portables use.)

QUOTE
If you don't mind about illegality, I suggest using AACenc, since it has good quality and is free.


This may have been addressed a million times elsewhere, but how is AACenc illegal? Is this the encoder used with PsytelDrop? I have it but don't recall where I downloaded it. Didn't realize it was illegal. (Warez?) unsure.gif Or is it just a licensing issue?


--------------------
Folding@Home HydrogenAudio.org Team ID# 32639
http://folding.stanford.edu/
Go to the top of the page
+Quote Post
ilikedirtthe2nd
post Jun 16 2003, 08:50
Post #4





Group: Members
Posts: 470
Joined: 26-October 01
From: Germany
Member No.: 352



QUOTE (rjamorim @ Jun 16 2003 - 07:12 AM)
(Also, info on the 41_30sec sample is welcome)

the sample is from the song "#41" by the dave matthews band from the album crash.

allmusic link: http://www.allmusic.com/cg/amg.dll?p=amg&u...l=A4c6tk6dxqkrf

regards; ilikedirt
Go to the top of the page
+Quote Post
dev0
post Jun 16 2003, 08:54
Post #5





Group: Developer
Posts: 1679
Joined: 23-December 01
From: Germany
Member No.: 731



Very interesting, especially considering how bad Dolby did in the 64kbps test.
Overall I was quite suprised by the high quality of all codecs except for FAAC, but there definetly is still room for massive improvments. None of the codecs was transparent to a degree where I'd consider using it for archival purposes, but for portable/casual use, which is the main field of usage at 128kbps anyway, AAC looks very interesting.

Thanks to Roberto and everyone else, who helped and participated, for investing time and work on this test. I can't wait for the next test to see how AAC compares to Vorbis, Musepack and MP3...

dev0

This post has been edited by dev0: Jun 16 2003, 08:56


--------------------
"To understand me, you'll have to swallow a world." Or maybe your words.
Go to the top of the page
+Quote Post
den
post Jun 16 2003, 08:55
Post #6





Group: Members
Posts: 315
Joined: 18-February 03
From: Perth, Western Australia
Member No.: 5050



I am also interested to see how aacenc -streaming compares to QT. I was very surprised when I was informed which samples were which encoder after I submitted my test results. Psytel -streaming in my own tests has been quite impressive for the bitrate, and to have it fixed as CBR in this test was not really letting it work at its best. I also understand why this testing decision was made however, for the sake of fairness.

As for FAAC vs mp3 128 kbit, I haven't done a direct back to back comparison on identical samples, but some of the FAAC samples in this test were really bad, as can be seen from the charts. I'm relatively new to codec listening tests, but in some FAAC samples, the voices/instruments actually sounded different in their tonality from the others, let alone pre-echo or other more typical codec introduced artifacts. blink.gif

Den.


--------------------
Den
My blog - http://www.iinet.net.au/~den
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Jun 16 2003, 09:24
Post #7


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



QUOTE
I am also interested to see how aacenc -streaming compares to QT. I was very surprised when I was informed which samples were which encoder after I submitted my test results. Psytel -streaming in my own tests has been quite impressive for the bitrate, and to have it fixed as CBR in this test was not really letting it work at its best. I also understand why this testing decision was made however, for the sake of fairness.


There is a ongoing improvement to Ahead Nero "streaming" preset as well, some changes already were made (still not out) as long as some grouping improvements and bug fixes. This will probably be available in the next web version of the Nero.
Go to the top of the page
+Quote Post
Dalkus
post Jun 16 2003, 09:32
Post #8





Group: Members
Posts: 34
Joined: 16-April 03
From: WI
Member No.: 6016



Interesting results. Does this test say anything about how the decoders are doing with VBR files in that bit rate area? (i.e. the encoders that are able to produce VBR files). I'm thinking if 128 CBR isn't good enough for portable use, maybe ~128 VBR is?
Go to the top of the page
+Quote Post
guruboolez
post Jun 16 2003, 09:36
Post #9





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (den @ Jun 16 2003 - 08:55 AM)
I am also interested to see how aacenc -streaming compares to QT.

I did some preliminary tests, and I compared blindly QT 6.3 to PsyTEL & Nero -streaming. Tested on 6 samples of this test.
Difference is impressive. Most audible one is the higher lowpass, which provide a richer sound than QT (appreciable on Atrain and 41_30), and less artifacts in some case. Now, PsyTEL & Ahead Nero AAC encodings are close to QT (for my taste), and sometime better than QT (41_30 for exemple). Unfortunately, I didn't have too much time last week to test it further. And I broke my headphone. The new one is here now, and I have some time the 2 next days smile.gif

My results are here :

http://membres.lycos.fr/guruboolez/AUDIO/a...128/tableau.txt

and for VBR :
http://membres.lycos.fr/guruboolez/AUDIO/a...tableau_vbr.txt
Go to the top of the page
+Quote Post
bawjaws
post Jun 16 2003, 11:11
Post #10





Group: Members
Posts: 173
Joined: 10-December 02
Member No.: 4043



minor nitpick with track info:

It should be "You've Got the Love" by The Source feat. Candi Staton (Although a popular dance tune and available on many compilations I don't know if the track was ever featured on an actual album by The Source)

-- a happy Mac user B) off to harass Apple for VBR capabities
Go to the top of the page
+Quote Post
Paspro
post Jun 16 2003, 11:32
Post #11





Group: Members
Posts: 103
Joined: 1-January 03
From: Athens, Greece
Member No.: 4350



What I find strange is the difference between the Sorenson (FhG Pro codec) and the QT (Dolby codec) results. I have the impression that the Dolby codec is based on the FhG codec with optimisations for speed rather than quality. Still, according to my experience, QuickTime Pro 6.3 and Sorenson Squeeze 3 take almost the same amount of time to encode an audio file in AAC, when the quality setting for QT is set to the highest setting. Could it be that this setting disables all options which reduce the quality of the encoding in favour of speed?

Another interesting point is the following link from Dolby Labs:

http://www.dolbylabs.com/DolbyAAC/FAQ/

which contains the following note:

QUOTE
How does Dolby AAC differ from AAC?
Dolby AAC consists of a standard MPEG-AAC core plus proprietary enhancements that enable a wider frequency range and better audio quality at lower bit rates.


So, what is Dolby's AAC codec? An improved FhG codec for both quality and speed, as the listening test results indicate?

This post has been edited by Paspro: Jun 16 2003, 11:33
Go to the top of the page
+Quote Post
guruboolez
post Jun 16 2003, 11:37
Post #12





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (Paspro @ Jun 16 2003 - 11:32 AM)
Still, according to my experience, QuickTime Pro 6.3 and Sorenson Squeeze 3 take almost the same amount of time to encode an audio file in AAC, when the quality setting for QT is set to the highest setting.

We didn't have the same experience. According to my timings, QT is one of the fastest AAC codec, and Sorenson the slowest with PsyTEL :

http://membres.lycos.fr/guruboolez/AUDIO/a...ac128/speed.txt
Go to the top of the page
+Quote Post
Paspro
post Jun 16 2003, 11:58
Post #13





Group: Members
Posts: 103
Joined: 1-January 03
From: Athens, Greece
Member No.: 4350



QUOTE (guruboolez @ Jun 16 2003 - 02:37 AM)
QUOTE (Paspro @ Jun 16 2003 - 11:32 AM)
Still, according to my experience, QuickTime Pro 6.3 and Sorenson Squeeze 3 take almost the same amount of time to encode an audio file in AAC, when the quality setting for QT is set to the highest setting.

We didn't have the same experience. According to my timings, QT is one of the fastest AAC codec, and Sorenson the slowest with PsyTEL :

http://membres.lycos.fr/guruboolez/AUDIO/a...ac128/speed.txt

Hmm... I am a MacOS X user and on that platform QT6.3 (with highest quality setting) and Sorenson take almost the same amount of time to encode at 128Kbps.
Go to the top of the page
+Quote Post
guruboolez
post Jun 16 2003, 12:06
Post #14





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (Paspro @ Jun 16 2003 - 11:58 AM)
Hmm...  I am a MacOS X user and on that platform QT6.3 (with highest quality setting) and Sorenson take almost the same amount of time to encode at 128Kbps.

Interesting. Therefore, for 95% of the users in this world, QT is three time faster than Sorenson (and a lot cheaper).
What is interesting to note is that QT is the absolute winner of the test, and in the same time, one of the fastest codec (FAAC is faster on my Duron, but quality isn't as good). Unfortunately, the interface is not the prettiest I saw... Hope this will change, by implemeting soon the same codec in iTunes.
Go to the top of the page
+Quote Post
ff123
post Jun 16 2003, 12:08
Post #15


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



Nice test, Roberto.

A few comments:

1. I think classical and jazz could have been better represented.

2. It should be noted somewhere, probably in the recommendations section, that this was a CBR test only, and that Nero and Psytel also have VBR modes, which perform better, according to Guruboolez. You might link to his listening results.

3. The crack about people advertising for FAAC is unneccessary. and doesn't help you win over a certain enthusiast to participate in your next test. smile.gif

4. You mention that you used an ANOVA analysis, but maybe you should also mention that this is different from what the 64 kbit/s test used. The similar presentation format might make people think that all the analysis was identical. The difference is mainly one about risk. The ANOVA / Fisher LSD method is more at risk for falsely identifying differences between codecs. On the other hand, it's more sensitive than the Tukey HSD.

5. I'm still uncomfortable with the squishy way that a summary graph is constructed. But since I can't think of a better way, and people have a need to see things in one, concise picture, I suppose it must be that way.

6. In the more detailed pages to follow, I'd like to see some mention about how a time misalignment of only 25 msec spoiled at least one result. Also, I'd like to see some mention of the results you threw out for rating the original less than 5.


Some ideas for a future test:

1. Perhaps another call for samples -- classical and jazz samples -- would be profitable.

2. You might think about adding at least one anchor sample -- a lowpassed version of the original, a la MUSHRA. This can be done with a small filesize penalty using Sox. That would help to keep the ratings in perspective.

3. Verifying VBR average bitrates: I think that this task could be split up among several people, each encoding whole albums with all codecs.

ff123

Edit: Oh, and if iTunes doesn't use the same codec that you used for this test, I would make some mention of that fact too.

Edit2: The next test you'll probably want to be sure to check for level (volume) differences too.

This post has been edited by ff123: Jun 16 2003, 12:14
Go to the top of the page
+Quote Post
Garf
post Jun 16 2003, 12:13
Post #16


Server Admin


Group: Admin
Posts: 4885
Joined: 24-September 01
Member No.: 13



ff123,

What do you think about 'eyeballing' the results to get the ranks. Wouldn't a straight ranking be more solid and not necessarily less powerfull? I see the eyeballing was done in the 64kbps test also.

What about a bootstrap analysis of the results. Possible?
Go to the top of the page
+Quote Post
Paspro
post Jun 16 2003, 12:13
Post #17





Group: Members
Posts: 103
Joined: 1-January 03
From: Athens, Greece
Member No.: 4350



QUOTE (guruboolez @ Jun 16 2003 - 03:06 AM)
QUOTE (Paspro @ Jun 16 2003 - 11:58 AM)
Hmm...  I am a MacOS X user and on that platform QT6.3 (with highest quality setting) and Sorenson take almost the same amount of time to encode at 128Kbps.

Interesting. Therefore, for 95% of the users in this world, QT is three time faster than Sorenson (and a lot cheaper).
What is interesting to note is that QT is the absolute winner of the test, and in the same time, one of the fastest codec (FAAC is faster on my Duron, but quality isn't as good). Unfortunately, the interface is not the prettiest I saw... Hope this will change, by implemeting soon the same codec in iTunes.

This is true. The QuickTime interface does not allow the batch processing of files and therefore it is not easy to use it for mass encoding of CDs. Also note that on MacOS X the iTunes application does not allow the use of the highest quality setting of the AAC codec and therefore the only option left for someone is to use the Ovolab AAChoo front-end

http://www.ovolab.com/

for batch encoding at any quality setting. This is what I use for my iPod encodings and it works well. Still, I hope that the next version of iTunes will allow the setting of the encoding quality. I also hope that the next QuickTime revision will support VBR AAC encodings...

This post has been edited by Paspro: Jun 16 2003, 12:14
Go to the top of the page
+Quote Post
ff123
post Jun 16 2003, 12:26
Post #18


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



QUOTE (Garf @ Jun 16 2003 - 03:13 AM)
ff123,

What do you think about 'eyeballing' the results to get the ranks. Wouldn't a straight ranking be more solid and not necessarily less powerfull? I see the eyeballing was done in the 64kbps test also.

What about a bootstrap analysis of the results. Possible?

The problem I was having with just using the data directly to get an overall summary is the idea that one sample might have a greater influence on the overall results than another. For example if one sample had a clear winner with a mean of 4.5, but another sample had a different clear winner with a mean of 2.5, I was thinking that the first sample would be given more weight in the overall results.

However, perhaps that's not really an issue when using a "blocked" analysis, which is supposed to take care of such things. Different listeners have varying internal quality scales. So I suppose that's analagous to the different samples having varying difficulty levels.

In that case, I would just try the exact same ANOVA / Fisher LSD on the data directly.

Problem with bootstrap is that it doesn't easily give you the nice 95% confidence intervals for the graphs. Also, it's similar to the Tukey HSD in being more conservative with risk (and less sensitive).

ff123

Edit: Roberto, don't forget to include the ATrain and Layla numbers when you do this analysis

This post has been edited by ff123: Jun 16 2003, 12:35
Go to the top of the page
+Quote Post
Cobra
post Jun 16 2003, 12:55
Post #19





Group: Banned
Posts: 344
Joined: 14-September 02
Member No.: 3365



Maybe compare AAC and Ogg Vorbis (both GT3b1 and latest CVS) @ 128kbps ?
Go to the top of the page
+Quote Post
dev0
post Jun 16 2003, 13:01
Post #20





Group: Developer
Posts: 1679
Joined: 23-December 01
From: Germany
Member No.: 731



QUOTE (Cobra @ Jun 16 2003 - 12:55 PM)
Maybe compare AAC and Ogg Vorbis (both GT3b1 and latest CVS) @ 128kbps ?

Learn to read! It was mentioned in the original announcement, that the winner of the AAC test will compete against MP3, Musepack and Vorbis. I'd probably try GT2's 128kbps mode too.

dev0


--------------------
"To understand me, you'll have to swallow a world." Or maybe your words.
Go to the top of the page
+Quote Post
Garf
post Jun 16 2003, 13:09
Post #21


Server Admin


Group: Admin
Posts: 4885
Joined: 24-September 01
Member No.: 13



I would definetly _not_ use any of the tuned Vorbis encoders for 128kbps. Just plain 1.0 -q4 should be best.
Go to the top of the page
+Quote Post
guruboolez
post Jun 16 2003, 13:28
Post #22





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (Garf @ Jun 16 2003 - 01:09 PM)
I would definetly _not_ use any of the tuned Vorbis encoders for 128kbps. Just plain 1.0 -q4 should be best.

I'm not agree. On most samples (i.e. not problematic), Ogg GT2 128 seems to be more transparent, due to the hiss/noise issue of 1.0 release.
With killer or problem sample, 1.0 may win (not sure).
Go to the top of the page
+Quote Post
feces1223
post Jun 16 2003, 13:33
Post #23





Group: Banned
Posts: 154
Joined: 30-March 03
Member No.: 5756



Yeah I'd Never Expect QT to come in as top cuz i thought there audio support was crap. The whole domination over movie trailers is pretty good though (QT trailers dominate WMP or REAL). Do you think Apple Store rips their albums using Quicktime? And which version and up supports AAC ripping (obviously Pro)?
Go to the top of the page
+Quote Post
Garf
post Jun 16 2003, 13:35
Post #24


Server Admin


Group: Admin
Posts: 4885
Joined: 24-September 01
Member No.: 13



QUOTE (guruboolez @ Jun 16 2003 - 02:28 PM)
QUOTE (Garf @ Jun 16 2003 - 01:09 PM)
I would definetly _not_ use any of the tuned Vorbis encoders for 128kbps. Just plain 1.0 -q4 should be best.

I'm not agree. On most samples (i.e. not problematic), Ogg GT2 128 seems to be more transparent, due to the hiss/noise issue of 1.0 release.
With killer or problem sample, 1.0 may win (not sure).

Yes, but doesn't GT2's 128kbps mode give more than 128k on average?

GT2 128kbps is Vorbis RC2 128kbps with lossless channel coupling

I would be surprised if that's still state-of-the-art as far as Vorbis at the most common bitrate goes.
Go to the top of the page
+Quote Post
guruboolez
post Jun 16 2003, 14:07
Post #25





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (Garf @ Jun 16 2003 - 01:35 PM)
Yes, but doesn't GT2's 128kbps mode give more than 128k on average ?

Don't really know. I haven't played a lot with vorbis, especially at this bitrate.
I encoded the 12 samples with GT2 (bitrate is of course not representative of a general behaviour) :

CODE
001. 41_30sec :       150 kbps
002. ATrain :         144 kbps
003. BeautySlept :    123 kbps
004. Blackwater :     130 kbps
005. death2 :         106 kbps
006. FloorEssence :   132 kbps
007. Layla :          152 kbps
008. LifeShatters :   138 kbps
009. MidnightVoyage : 148 kbps
010. thear1 :         142 kbps
011. TheSource :      128 kbps
012. Waiting :        148 kbps



==> 136.75 kbps (according to foobar2000)
Go to the top of the page
+Quote Post

6 Pages V   1 2 3 > » 
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 19th September 2014 - 18:21