IPB

Welcome Guest ( Log In | Register )

6 Pages V  < 1 2 3 4 > »   
Reply to this topicStart new topic
AAC at 128kbps test - FINISHED
dev0
post Jun 16 2003, 15:15
Post #26





Group: Developer
Posts: 1679
Joined: 23-December 01
From: Germany
Member No.: 731



I can only support Guru's comment: GT2's 128kbps mode seems to be the most transparent Vorbis encoder at around 130kbps at least in the few tests I've done. Maybe there should be some extra testing done about GT2 vs. Post-1.0-CVS Vorbis at 128kbps before deciding which one to use for the big 128kbps codec shoot out.

dev0


--------------------
"To understand me, you'll have to swallow a world." Or maybe your words.
Go to the top of the page
+Quote Post
guruboolez
post Jun 16 2003, 15:41
Post #27





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (dev0 @ Jun 16 2003 - 03:15 PM)
Maybe there should be some extra testing done about GT2 vs. Post-1.0-CVS Vorbis at 128kbps before deciding which one to use for the big 128kbps codec shoot out.

dev0

I can't imagine a big listening test in july 2003, including an early codec, dated from november 2001. Most people will use a CVS or PostCVS (never heard anything different) version for their ~128 encodings.

Nevertheless, a dedicated listening test between GT2 and 1.0 should be started. If conclusions are in favor of the old codec version, it may convinced Xiph people of some flaws in their 'final' codec. The noise issue is acceptable at low bitrate (50-90 kbps), and I prefered some hiss that metallic distorsion. But at ~120 kbps, I prefer some occasional distorsion than a continuous noise, perceptible on most music. Especially when I tested QT encondings, hiss-free and with very few artifacts.

This post has been edited by guruboolez: Jun 16 2003, 15:42
Go to the top of the page
+Quote Post
treech
post Jun 16 2003, 16:06
Post #28





Group: Members
Posts: 208
Joined: 28-May 02
From: Helsingborg
Member No.: 2147



AACenc = Sorenson ?
Go to the top of the page
+Quote Post
guruboolez
post Jun 16 2003, 16:08
Post #29





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (treech @ Jun 16 2003 - 04:06 PM)
AACenc = Sorenson ?

PsyTEL AACenc.exe version 2.15
Go to the top of the page
+Quote Post
treech
post Jun 16 2003, 16:20
Post #30





Group: Members
Posts: 208
Joined: 28-May 02
From: Helsingborg
Member No.: 2147



ahh, ok thanks

also, would interresting to se if/how much worse psytel fastenc is compared to the normal psytel...
Go to the top of the page
+Quote Post
ff123
post Jun 16 2003, 16:25
Post #31


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



Quick and dirty new analysis, in response to complaints about the "eyeball" method of ranking

Substituting all 4.5's for the two samples in which means are not shown, I quickly came up with the following graph for the overall results, using ANOVA / Fisher LSD instead of the "eyeball" ranking method:

http://ff123.net/export/aac.png

The rankings are about the same as before, with QT the clear winner, FAAC the clear loser, and the rest tied. However, this method has the advantage of showing how close the codecs are to the reference score of 5.0

In a future test, I recommend using an anchor (filtered version of the original) to put things in better perspective.

Edit: the graph label should be corrected to read "Rating" instead of "Ranking"

This post has been edited by ff123: Jun 16 2003, 16:53
Go to the top of the page
+Quote Post
guruboolez
post Jun 16 2003, 16:36
Post #32





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



Back to some results.

I'm really surprised to see BeautySlept notation. I tested this sample in two test, this one and a preliminary one.

CODE
             General    Personal    Previous
Quicktime     4.42        3.7          4.2
Ahead Nero    3.81        1.4          2.1 *
PsyTEL        4.25        2.9          1.8
Sorenson      4.26        2.5          2.5
FAAC          3.92        2.0          --- **

* Ahead MP4 Codec was an older version
** FAAC was not tested previously

I systematically found all encoding, except QuickTime, more than annoying. And I can't bear Ahead codec distorsions ! Even with VBR (-streaming) , Ahead was rated 2.5 and PsyTEL 3.5 (QT = 5.0 on this test !).
Can't really explain why. It is because I'm most familiar with harpsichord, and can't bear any distorsions ? Or maybe headphone issues ? In fact, I tried to perform this test with a set of poor earbuds (Sennheiser MX-500), began with BeautySlept sample, and found all codecs, including Ahead & FAAC, to be near transparency ! I gave up on this result... With a good set of headphone, the distorsion was not only easily perceptible, but awfully annoying.

P.S. According to the general conclusion, BeautySlept is one of the two samples where FAAC provide a better sound than Nero encoding (3.81 vs 3.92 - second one is LifeShatters, 3.88 vs 3.89)

This post has been edited by guruboolez: Jun 16 2003, 16:43
Go to the top of the page
+Quote Post
ckjnigel
post Jun 16 2003, 16:53
Post #33





Group: Members
Posts: 218
Joined: 12-October 01
Member No.: 278



Thanks to Roberto and know that my admiration continues to grow. I think this is fascinating.
1) Any speculation as to why ATrain and Layla were so easy for AAC? Surely they were chosen because the samples were challenging to MP3, Ogg and MPC. (I'm guessing the next test will address issues of what samples are handled relatively better or worse by the codecs.)
2) What encoder does AOL use for the AAC tracks streamed via Radio@AOL?
3) One guesses that Apple is devoting considerable resources to further development. Are Ahead and/or Sorenson doing sufficient work to make one expect substantial improvements in their offerings? [Afterwards addition: OK - Ivan's quick response and comments suggest Ahead is -- tnx, Ivan!] Is there another developer readying an offering? What prevents Dolby from doing so?
4) Do any or all of these codecs provide enhanced capacities for "digital rights management" vs. what wma already has and what mpc and ogg can offer? Or is that a matter dependent solely on the OS platform?
Adding an off-topic rant, I wish there were a way to build support for MPC. "As flies to wanton boys are we to the gods, they kill us for the sport," is my impression of the corporate Olympians' treatment of the public. "The public be damned, we'll tell them what they want," is the way media and software conglomerates are handling the evolution of codecs.

This post has been edited by ckjnigel: Jun 16 2003, 18:18
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Jun 16 2003, 17:06
Post #34


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



Guruboolez, ask Menno for the internal NERO plug-in update - I think I found the reason for the BeautySlept problem, and it should be much better now -

File was ended up heavily undercoded at 112 kbps - clearly a CBR bug (which is being fixed now)

QUOTE
1) Any speculation as to why ATrain and Layla were so easy for AAC? Surely they were chosen because the samples were challenging to MP3, Ogg and MPC. (I'm guessng the next test will address issues of what samples are handled relatively better or worse by the codecs.)


Atrain has relatively low perceptual entropy, making it easy to encode at 128 kbps for AAC - also, most passages are "noisy" which gives more masking abilities for the psych model. Same goes for layla (noisy)
Go to the top of the page
+Quote Post
rjamorim
post Jun 16 2003, 18:21
Post #35


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (ChrisGranger @ Jun 16 2003 - 04:39 AM)
Just a little typo in the last line: "Sorenson is good, but it's price is prohibitive." It should be "its" which is possessive.

Fixed. Thank-you very much.

QUOTE
This may have been addressed a million times elsewhere, but how is AACenc illegal? Is this the encoder used with PsytelDrop? I have it but don't recall where I downloaded it. Didn't realize it was illegal. (Warez?)  unsure.gif Or is it just a licensing issue?


Only licensing issues. It's not warez, or pirate, because the copyright owner (Ivan) allowed it to be distributed.


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
rjamorim
post Jun 16 2003, 18:24
Post #36


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (ilikedirtthe2nd @ Jun 16 2003 - 04:50 AM)
the sample is from the song "#41" by the dave matthews band from the album crash.

allmusic link: http://www.allmusic.com/cg/amg.dll?p=amg&u...l=A4c6tk6dxqkrf

regards; ilikedirt

Thanks.

2 more pieces of info: who submitted it (if it's known), and what's the style?


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
rjamorim
post Jun 16 2003, 18:32
Post #37


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (bawjaws @ Jun 16 2003 - 07:11 AM)
minor nitpick with track info:

It should be "You've Got the Love" by The Source feat. Candi Staton (Although a popular dance tune and available on many compilations I don't know if the track was ever featured on an actual album by The Source)

Fixed (i think) smile.gif

Thanks a lot.


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
rjamorim
post Jun 16 2003, 18:35
Post #38


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (Paspro @ Jun 16 2003 - 07:32 AM)
So, what is Dolby's AAC codec? An improved FhG codec for both quality and speed, as the listening test results indicate?

It's a FhG codec improved for speed. And, miraculously, it also improves quality while at it. :B


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
rjamorim
post Jun 16 2003, 19:03
Post #39


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



[quote=ff123,Jun 16 2003 - 08:08 AM] 1. I think classical and jazz could have been better represented. [/quote]
Well, Lizst and Bach were removed because people guessed they would be too easy, and therefore not significative.

Besides, we needed to make space for problem samples, and JohnV wouldn't accept a test with more than 12 samples. wink.gif

[quote]2.  It should be noted somewhere, probably in the recommendations section, that this was a CBR test only, and that Nero and Psytel also have VBR modes, which perform better, according to Guruboolez.  You might link to his listening results.[/quote]

Addressed. Thanks.

[quote]3.  The crack about people advertising for FAAC is unneccessary. and doesn't help you win over a certain enthusiast to participate in your next test. smile.gif[/quote]

He didn't participate in this one already. :-P

[quote]4.  You mention that you used an ANOVA analysis, but maybe you should also mention that this is different from what the 64 kbit/s test used.  The similar presentation format might make people think that all the analysis was identical.  The difference is mainly one about risk.  The ANOVA / Fisher LSD method is more at risk for falsely identifying differences between codecs.  On the other hand, it's more sensitive than the Tukey HSD.[/quote]

Addressed

[quote]5.  I'm still uncomfortable with the squishy way that a summary graph is constructed.  But since I can't think of a better way, and people have a need to see things in one, concise picture, I suppose it must be that way.[/quote]

Well, I have no clue about statistics. If you have any idea to fix that, please inform me.

[quote]6.  In the more detailed pages to follow, I'd like to see some mention about how a time misalignment of only 25 msec spoiled at least one result.  Also, I'd like to see some mention of the results you threw out for rating the original less than 5.[/quote]

OK, I will do that.

Of course, I won't make public the names of the participants that rated the reference. That wouldn't be nice to them, I believe.

In your test, you had people rating the reference too?


[quote]1.  Perhaps another call for samples -- classical and jazz samples -- would be profitable.[/quote]

Well, that's OK, but I planned to use the same test suite, even with the samples that ended up "too transparent". Else, if I change the suite too much, whiners will say there's no significance between the first test and the extension. "Who knows if QuickTime would win in this new suite? neener-neener"

[quote]2.  You might think about adding at least one anchor sample -- a lowpassed version of the original, a la MUSHRA.  This can be done with a small filesize penalty using Sox.  That would help to keep the ratings in perspective.[/quote]

OK, I think I'll do that. Thanks for the idea.

[quote]3.  Verifying VBR average bitrates:  I think that this task could be split up among several people, each encoding whole albums with all codecs.[/quote]

Yes, I plan to start a taskforce on this later this month.

[quote]Edit:  Oh, and if iTunes doesn't use the same codec that you used for this test, I would make some mention of that fact too.[/quote]

Done

[quote]Edit2:  The next test you'll probably want to be sure to check for level (volume) differences too.[/quote]

OK, I will.

Regards;

Roberto.


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
NickSD
post Jun 16 2003, 19:06
Post #40





Group: Members
Posts: 135
Joined: 20-December 01
Member No.: 702



QUOTE (rjamorim @ Jun 16 2003 - 10:24 AM)
Thanks.

2 more pieces of info: who submitted it (if it's known), and what's the style?

I don't know who submitted it, but the style would be something along the lines of jazz-rock. Dave Matthews Band is very hard to categorize... (Incredible live band, by the way.)

Nick
Go to the top of the page
+Quote Post
Garf
post Jun 16 2003, 19:09
Post #41


Server Admin


Group: Admin
Posts: 4884
Joined: 24-September 01
Member No.: 13



QUOTE (rjamorim @ Jun 16 2003 - 08:03 PM)
Well, that's OK, but I planned to use the same test suite, even with the samples that ended up "too transparent". Else, if I change the suite too much, whiners will say there's no significance between the first test and the extension. "Who knows if QuickTime would win in this new suite? neener-neener"

That argument simply doesn't work at all.

If you believe another set of samples would produce a significant performance difference between the AAC codecs compared to what we've seen here, it follows directly from that the next test would be highly unfairly biased towards AAC if you did use the same samples.

This post has been edited by Garf: Jun 16 2003, 19:10
Go to the top of the page
+Quote Post
niktheblak
post Jun 16 2003, 19:36
Post #42





Group: Members (Donating)
Posts: 302
Joined: 3-October 01
From: Finland
Member No.: 188



QUOTE (guruboolez @ Jun 16 2003 - 05:36 PM)
Can't really explain why. It is because I'm most familiar with harpsichord, and can't bear any distorsions ?

Cradle Of Filth's harpsichord naturally comes from a synthesizer so there may be differences, whether surprising or subtle, with real harpsichord music.

I have to ask before I lose my mind, did the BeautySlept sample originate from me (in this thread) or is it a funny coincidence smile.gif
Go to the top of the page
+Quote Post
rjamorim
post Jun 16 2003, 19:42
Post #43


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (ckjnigel @ Jun 16 2003 - 12:53 PM)
1) Any speculation as to why ATrain and Layla were so easy for AAC?  Surely they were chosen because the samples were challenging to MP3, Ogg and MPC.  (I'm guessing the next test will address issues of what samples are handled relatively better or worse by the codecs.)

No, they were chosen because they were featured in ff123's 64kbps test. smile.gif

I will probably ditch them for the extension test though (thanks for the clarification, Garf)

QUOTE
2) What encoder does AOL use for the AAC tracks streamed via Radio@AOL?


No idea.

QUOTE
3) One guesses that Apple is devoting considerable resources to further development.  Are Ahead and/or Sorenson doing sufficient work to make one expect substantial improvements in their offerings? [Afterwards addition: OK - Ivan's quick response and comments suggest Ahead is -- tnx, Ivan!] Is there another developer readying an offering? What prevents Dolby from doing so?


There's another developer, that actually offered to send me a demo encoder that I can use in some future listening test, but I don't know if I can decline informations now.

What prevents dolby from doing what?

QUOTE
4) Do any or all of these codecs provide enhanced capacities for "digital rights management" vs. what wma already has  and what mpc and ogg can offer? Or is that a matter dependent solely on the OS platform?
Adding an off-topic rant, I wish there were a way to build support  for MPC.  "As flies to wanton boys are we to the gods, they kill us for the sport," is my impression of the corporate Olympians' treatment of the public.  "The public be damned, we'll tell them what they want," is the way media and software conglomerates are handling the evolution of codecs.


There's no standardized DRM system for AAC. And no, none of these encoders offer the option of DRM'ing your AACs, differently from WMA/Windows Media Encoder.

Regards;

Roberto.

@NickSD: Thanks for the info.

This post has been edited by rjamorim: Jun 16 2003, 19:44


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
guruboolez
post Jun 16 2003, 19:50
Post #44





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (niktheblak @ Jun 16 2003 - 07:36 PM)
QUOTE (guruboolez @ Jun 16 2003 - 05:36 PM)
Can't really explain why. It is because I'm most familiar with harpsichord, and can't bear any distorsions ?

Cradle Of Filth's harpsichord naturally comes from a synthesizer so there may be differences, whether surprising or subtle, with real harpsichord music.

I was usually embarassed by the synthetic consistency of this harpsichord (and by the metronomic performance of the player). Thank you for this confirmation smile.gif
Here is a real harpsichord sample. Bitrate is higher than with BeautySlept (265 kbps with --standard and 1.15r). It's a real killer too for most codec, if you want to reach 128 kbps and not more. I strongly suggest anyone to test it with WMA9 (PRO), incredibly transparent at 128 kbps compared to all other format/codecs I heard.

http://membres.lycos.fr/guruboolez/AUDIO/s...chpsichord.flac
Go to the top of the page
+Quote Post
ff123
post Jun 16 2003, 20:43
Post #45


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



QUOTE (rjamorim @ Jun 16 2003 - 10:03 AM)
QUOTE
5.  I'm still uncomfortable with the squishy way that a summary graph is constructed.  But since I can't think of a better way, and people have a need to see things in one, concise picture, I suppose it must be that way.


Well, I have no clue about statistics. If you have any idea to fix that, please inform me.

I think the best way is to just use the same procedure that you did for each sample. But you'll have to calculate the mean scores for ATrain and Layla first (easiest to just use Excel for that).

ff123
Go to the top of the page
+Quote Post
ff123
post Jun 16 2003, 20:49
Post #46


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



Regarding 41_30sec.wav

QUOTE (rjamorim @ Jun 16 2003 - 09:24 AM)
2 more pieces of info: who submitted it (if it's known), and what's the style?

Originally submitted by Filburt

style: rock

This post has been edited by ff123: Jun 16 2003, 20:50
Go to the top of the page
+Quote Post
ErikS
post Jun 16 2003, 21:34
Post #47





Group: Members
Posts: 757
Joined: 8-October 01
Member No.: 247



Does ANOVA take into consideration the varying scales that people use when they grade the test samples? For example someone who thinks that the goal of the codec is to achive total transparancy may grade everything between 1 and 3 when someone else who doesn't care that much would give grades between 4 and 5 to the same set.

So could someone very briefly explain how ANOVA works (or post some good links)? Which types of data sets is it possible to use ANOVA on?
Go to the top of the page
+Quote Post
Agent86
post Jun 16 2003, 21:58
Post #48





Group: Members
Posts: 55
Joined: 2-October 01
Member No.: 162



Does anyone know anything about the AAC codec that XM is using to broadcast its satellite radio?

XM AAC Information

It apppears as though it is heavily customized, but I wonder if it originally came from anything used in this listening test.

- Agent 86
Go to the top of the page
+Quote Post
rjamorim
post Jun 16 2003, 22:13
Post #49


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



XM is probably using gear from Thales
http://www.thales-bm.com/
Probably this baby.
It's the same gear most DRM stations are using.

Thales uses AAC encoder from FhG and SBR encoder from CT. Probably integerized versions.

This post has been edited by rjamorim: Jun 16 2003, 22:16


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
Differenciam
post Jun 16 2003, 22:48
Post #50





Group: Members
Posts: 453
Joined: 15-January 03
Member No.: 4577



Very interesting. smile.gif I'm quite disappointed in Nero, yet another reason I just keep re-using their trial. :roll: tongue.gif

I'm starting to wish my player supported AAC now. sad.gif
Go to the top of the page
+Quote Post

6 Pages V  < 1 2 3 4 > » 
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 29th August 2014 - 09:43