IPB

Welcome Guest ( Log In | Register )

2 Pages V   1 2 >  
Reply to this topicStart new topic
Ahead AAC VBR vs Vorbis aoTuV beta 3, at ~130 kbps with classical music
guruboolez
post Dec 9 2004, 08:26
Post #1





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



The excellent performances of newer Nero AAC encoder (2.9.998 “fast”) lead me to compare it with other strong competitors. Unfortunately, there are not so many potential challengers.

lame MP3: I’m using it every day, and though I like his new VBR mode (-V 5), I’m sure it’s not strong enough with my music to fully compete with the new AAC VBR encoder.

wma9 PRO: I’m fond of this encoder, working very well with classical music. I don’t know how perform the newest 9.1 encoder. Unfortunately, two years after the first release of WMA9PRO, there’s still apparently no dedicated portable player supporting it (except maybe telephone or PDA). The industry is still manufacturing electronic components and players for standard WMA, which is not a very competitive format.

ogg vorbis: since Aoyumi works, vorbis improved a lot (last year, with classical music, quality was behind lame ABR for my taste). aoTuV encoders (and now official 1.1) have drastically reduced the hiss/coarseness problems, which annoys me a lot with classical music (maybe other genres are affected too, I can’t really say). Unfortunately, the problem is only reduced, and not fully corrected. Nevertheless, I consider vorbis as one of the most robust format. Latest aoTuV beta 3, announcing an even lower coarseness level and a better pre-echo performance, could be a potential champion. I’d personally consider it a priori as the most dangerous challenger for strong AAC encoders.



I didn’t include wma9PRO in the following test. I don’t know that much this encoder: Is there one VBR profile close to ~130 kbps? Which one? Is it reliable, or is CBR/VBR2-pass a better solution? Answers are needed before including this format in a comparison test, and I don’t have time enough to answer them by myself.

Therefore, I’ve restricted the comparison to vorbis and Nero AAC. As encoder, I had some choice (official 1.1, megamix, aoTuV beta 2/3). I’ve privileged aoTuV encoder, because Aoyumi’s works are mainly focused on the most important vorbis problem (coarseness). I hesitated a moment between beta 2 and beta 3. I’ve finally opted for beta 3 without preliminary investigations: this test is a good occasion to see how reliable the new beta is, and if potential problems will handicap the format during this test, they will at last be detected, and probably quickly corrected.
I don’t know which vorbis setting would be the fairest to compete with Nero –internet. –q 4,00 is the roundest, and probably one of the most popular. In order to prevent (or at least to limit) some excessive difference, I’ve used –q 4,25. With my computer, I can’t obtain quickly a full and reliable bitrate table. But the comparison based on the tested samples is very satisfying:
- Nero encodings: 136,2 kbps
- aoTuV –q 4,25 encodings: 136,5 kbps
…hard to find closest settings :·)


The testing conditions (hardware, software, methodology…) are the same as those described in the global AAC comparison. Samples are also the same. One difference: I didn’t searched to ABX the difference between the encoded files and the reference ones (already done with all AAC encodings in previous test), but focused my efforts to a direct comparison between the two encoded files. Again, I’ve fixed a dual limit to the number of trials: eight if easy, sixteen trials if less easy. I have respected these limits, without any exception. First second of each sample was systematically discarded from listening.
Now, results:



vorbis aoTuV beta 3: Nice quality, but still coarseness issue. It’s fortunately limited, and with some samples, I simply can’t detect it (problem was systematically audible with older vorbis encoder). Very limited pre-echo problems: Aoyumi has apparently solved most issues in this area. However there is a deep ringing artefact with the organ (Bruhns) sample. I’ve in addition noticed a very strange problem, occurring with solo violin on two different samples (Avison first, and then Mozart-Sonata): it sounded like a noise-reduction processed recording, but additional hiss was audible on the same time. This paradox also happened to the old experimental Quantum Knot’s encoder (v. 3.5 aka vorbis “classical edition”). I don’t know: maybe Aoyumi did similar tuning to aoTuV beta 3? I had to investigate further…

compared to Nero AAC: better with harpsichord (note that aoTuV allocates even more bits for this instrument than Nero AAC!) on Couperin and Bach (excellent encoding) samples. I’m badly surprised by poor performances on Bayle sample: I’d expect from a VBR encoder to detect the difficulty (micro-attack). With aoTuV, both bitrate and quality are low… Noise/hiss/coarseness are like a signature with most samples, and often betray vorbis presence.


STATISTICS

You could try with the following table:
CODE
NERO    AOTUV
4,0    2,0
4,5    4,5
5,0    3,5
4,5    3,5
4,5    1,5
5,0    5,0
3,5    2,5
3,5    4,5
4,0    4,5
4,5    4,0
4,0    1,5
4,0    3,0
3,0    3,5
3,0    3,5
3,0    2,0


Conclusions are the same with ANOVA and Tukey parametric:

« NERO is better than AOTUV ».


• ABX logs are available here
• SAMPLES (11 MB) are available here. Keep them: I can’t keep them very long time online.

This post has been edited by guruboolez: Dec 29 2005, 22:20
Go to the top of the page
+Quote Post
guruboolez
post Dec 9 2004, 08:26
Post #2





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



In addition to this test, I’ve compared the performance of aoTuV beta3 to the previous beta 2 on two problem samples: Avison and Mozart-Sonata. To complete the investigation, I’ve also add QK 3.5 “classical edition” at –q 3,4 (this encoder need to lower the –q value for matching bitrate values).

Here are the results:

CODE
ABC/HR Version 0.9b, 30 August 2002
Testname:

1R = D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track01 (Avison - Concerto Grosso 30 sec).exe - quantum knot classical edition q 3,4.wav
2L = D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track01 (Avison - Concerto Grosso 30 sec).exe - aoTuV beta 3 q 4,25.wav
3R = D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track01 (Avison - Concerto Grosso 30 sec).exe - aoTuV beta 2 q 4,25.wav

---------------------------------------
General Comments:

---------------------------------------
1R File: D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track01 (Avison - Concerto Grosso 30 sec).exe - quantum knot classical edition q 3,4.wav
1R Rating: 1.0
1R Comment: Highly distorted violin. Hollow and empty.
---------------------------------------
2L File: D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track01 (Avison - Concerto Grosso 30 sec).exe - aoTuV beta 3 q 4,25.wav
2L Rating: 2.0
2L Comment: sound is hollow, like previous (1R) file, but impact is a bit lowered.
---------------------------------------
3R File: D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track01 (Avison - Concerto Grosso 30 sec).exe - aoTuV beta 2 q 4,25.wav
3R Rating: 3.5
3R Comment: a bit distorted.
---------------------------------------
ABX Results:


CODE
ABC/HR Version 0.9b, 30 August 2002
Testname:

1L = D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track11 (Mozart - Sonate KV 377).exe - aoTuV beta 3 q 4,25.wav
2L = D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track11 (Mozart - Sonate KV 377).exe - quantum knot classical edition q 3,4.wav
3R = D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track11 (Mozart - Sonate KV 377).exe - aoTuV beta 2 q 4,25.wav

---------------------------------------
General Comments:

---------------------------------------
1L File: D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track11 (Mozart - Sonate KV 377).exe - aoTuV beta 3 q 4,25.wav
1L Rating: 2.0
1L Comment: Empty sound. Musical matter is missing.
---------------------------------------
2L File: D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track11 (Mozart - Sonate KV 377).exe - quantum knot classical edition q 3,4.wav
2L Rating: 1.0
2L Comment: clearly the most affected by this hollow sounding.
---------------------------------------
3R File: D:\classical test december 2004\Nero AAC vs Vorbis aoTuV\further tests\Track11 (Mozart - Sonate KV 377).exe - aoTuV beta 2 q 4,25.wav
3R Rating: 2.5
3R Comment: very flat violin. Slightly better than 1L.
---------------------------------------
ABX Results:


Conclusion: aoTuV beta3 introduces similar artefacts on these two samples when compared to beta 2. The experimental “classical edition” have similar issue, but at higher level. Of course, it doesn’t necessary mean that beta 2 would be a better competitor than beta 3 against Nero AAC, but just that recent changes to aoTuV encoder have some negative consequences. I’ve already experimented regression of coarseness with beta 3 compared to beta 2.
Go to the top of the page
+Quote Post
JohnV
post Dec 9 2004, 11:14
Post #3





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



QUOTE (guruboolez @ Dec 9 2004, 10:26 AM)
The excellent performances of newer Nero AAC encoder (2.9.998 “fast”) lead me to compare it with other strong competitors.

The fast mode uses so called Scalefactor estimation mode, which hopefully will become the default very high quality mode after it has been refined.
After SFE and some other tweaks are working at full power, well, the LC-core should be extremely good, that is our goal.. smile.gif


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
john33
post Dec 9 2004, 11:25
Post #4


xcLame and OggDropXPd Developer


Group: Developer
Posts: 3761
Joined: 30-September 01
From: Bracknell, UK
Member No.: 111



Juha Laaksonheimo alias JohnV
Ahead Engineer


Congratulations, JohnV, HA seems to have been a very fertile recruiting ground for Ahead!!! biggrin.gif


--------------------
John
----------------------------------------------------------------
My compiles and utilities are at http://www.rarewares.org/
Go to the top of the page
+Quote Post
QuantumKnot
post Dec 9 2004, 13:18
Post #5





Group: Developer
Posts: 1245
Joined: 16-December 02
From: Australia
Member No.: 4097



Again, thanks for the insightful tests, guruboolez. They give Vorbis developers the perfect 'reality check' on their work. smile.gif

It seems classical music continues to be a weak point for Vorbis. *sigh*

Just as a sidenote, I found some free-time this week playing with some wavelet-based transient analysis (for block switching and dynamic impulse adjustment) for Vorbis 1.1. Nothing concrete to release yet, but the results look quite interesting. Expect an alpha of QKTune 2 sometime in the new year. cool.gif

This post has been edited by QuantumKnot: Dec 9 2004, 13:22
Go to the top of the page
+Quote Post
Aoyumi
post Dec 9 2004, 13:52
Post #6





Group: Members
Posts: 236
Joined: 14-January 04
From: Kanto, Japan
Member No.: 11215



@guruboolez,
Thank you for a test. I checked the clear problem with some samples.
I already recognize the cause. And I will release an experimental version in the near future, in order to solve or reduce this problem.

This post has been edited by Aoyumi: Dec 9 2004, 13:56
Go to the top of the page
+Quote Post
HotshotGG
post Dec 9 2004, 14:39
Post #7





Group: Members
Posts: 1593
Joined: 24-March 02
From: Revere, MA
Member No.: 1607



QUOTE
Just as a sidenote, I found some free-time this week playing with some wavelet-based transient analysis (for block switching and dynamic impulse adjustment) for Vorbis 1.1.


Care to eleborate? ;-D


--------------------
College student/IT Assistant
Go to the top of the page
+Quote Post
Redmond
post Dec 9 2004, 18:05
Post #8





Group: Members
Posts: 11
Joined: 9-December 04
Member No.: 18586



"I didn’t include wma9PRO in the following test. I don’t know that much this encoder: Is there one VBR profile close to ~130 kbps? Which one? Is it reliable, or is CBR/VBR2-pass a better solution? Answers are needed before including this format in a comparison test, and I don’t have time enough to answer them by myself."

1P CBR, 2P CBR, and 2P VBR have 128kbps modes. For this test, you should use either 2P CBR or 2P VBR.

This post has been edited by Redmond: Dec 9 2004, 18:06
Go to the top of the page
+Quote Post
rjamorim
post Dec 9 2004, 18:08
Post #9


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



Very interesting test. Thanks Guru.

Now we need a test comparing all musical styles. Who will be brave enough to stand up and conduce such test? (anyone who answers rjamorim will get socked in the eyes by yours truly tongue.gif )


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
Sebastian Mares
post Dec 9 2004, 19:26
Post #10





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



QUOTE (rjamorim @ Dec 9 2004, 06:08 PM)
Very interesting test. Thanks Guru.

Now we need a test comparing all musical styles. Who will be brave enough to stand up and conduce such test? (anyone who answers rjamorim will get socked in the eyes by yours truly tongue.gif )
*


Roberto Amorim should do it. (I did NOT say "rjamorim"! biggrin.gif)

Nah, joking... I would love to, but I don't have that much experience. sad.gif


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
Dologan
post Dec 9 2004, 21:47
Post #11





Group: Members (Donating)
Posts: 478
Joined: 22-November 01
From: United Kingdom
Member No.: 519



QUOTE (JohnV @ Dec 9 2004, 04:14 AM)
QUOTE (guruboolez @ Dec 9 2004, 10:26 AM)
The excellent performances of newer Nero AAC encoder (2.9.998 “fast”) lead me to compare it with other strong competitors.

The fast mode uses so called Scalefactor estimation mode, which hopefully will become the default very high quality mode after it has been refined.
After SFE and some other tweaks are working at full power, well, the LC-core should be extremely good, that is our goal.. smile.gif
*

Well, unrefined as it may be, it seems its quality is already higher than high. rolleyes.gif
Go to the top of the page
+Quote Post
JohnV
post Dec 9 2004, 22:13
Post #12





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



QUOTE (Dologan @ Dec 9 2004, 11:47 PM)
QUOTE (JohnV @ Dec 9 2004, 04:14 AM)
QUOTE (guruboolez @ Dec 9 2004, 10:26 AM)
The excellent performances of newer Nero AAC encoder (2.9.998 “fast”) lead me to compare it with other strong competitors.

The fast mode uses so called Scalefactor estimation mode, which hopefully will become the default very high quality mode after it has been refined.
After SFE and some other tweaks are working at full power, well, the LC-core should be extremely good, that is our goal.. smile.gif
*

Well, unrefined as it may be, it seems its quality is already higher than high. rolleyes.gif
*

There are still issues which Guru's test don't reveal, that's why it's not default yet.


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
QuantumKnot
post Dec 10 2004, 00:39
Post #13





Group: Developer
Posts: 1245
Joined: 16-December 02
From: Australia
Member No.: 4097



QUOTE (HotshotGG @ Dec 9 2004, 11:39 PM)
QUOTE
Just as a sidenote, I found some free-time this week playing with some wavelet-based transient analysis (for block switching and dynamic impulse adjustment) for Vorbis 1.1.


Care to eleborate? ;-D
*


Still early times but the results are interesting. That's all I'll say for now. wink.gif Note that I am not working on a quasi-Vorbis II. It is 100% pure Vorbis I.

btw. who else at HA has the experience (and time) to hold a listening test?
Go to the top of the page
+Quote Post
bond
post Dec 10 2004, 01:05
Post #14





Group: Members
Posts: 881
Joined: 11-October 02
Member No.: 3523



QUOTE (JohnV @ Dec 9 2004, 12:14 PM)
After SFE and some other tweaks are working at full power, well, the LC-core should be extremely good, that is our goal..

great stuff, neros aac encoder really needed this smile.gif


--------------------
I know, that I know nothing (Socrates)
Go to the top of the page
+Quote Post
DreamTactix291
post Dec 10 2004, 07:51
Post #15





Group: Members (Donating)
Posts: 552
Joined: 9-June 04
From: A place long since forgotten...
Member No.: 14572



Nero's AAC did really really well here. This was also a good test for aoTuV b3 as it has for the most part gone untested so far (but it is reall really new of course). I'm actually really impressed with Nero's AAC here.


--------------------
Nero AAC 1.5.1.0: -q0.45
Go to the top of the page
+Quote Post
guruboolez
post Dec 10 2004, 12:43
Post #16





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (JohnV @ Dec 9 2004, 10:13 PM)
There are still issues which Guru's test don't reveal, that's why it's not default yet.
*


I've just discovered the great performance of newer Nero AAC. I've encoded some additionnal tracks in order to perceive additional issues. Is it worth to send samples (to Garf, Ivan, Menno or you)? Or is it better (for you) to wait for the final release?
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Dec 10 2004, 13:18
Post #17


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



Hello,

First of all, guruboolez, many thanks for your tests - as always they are extremely useful to all of us developing codecs (please send any clips that you think that might be useful to us)

Also, I would like to clear out some unknowns about the "fast" mode -

"Fast" mode is using direct estimation of the scale-factor values - and it turned out that this method gives better results than traditional "two pass iteration loop" algorithm found in older generation of encoders.

This mode is still "beta" as many improvements are currently being done to both LC and SBR algorithms that should, we hope, result in significant improvement in quality of our codec.

I am also sorry for delay of the planned "V3.0" release of Nero AAC encoder - as we are all extremenly busy with the Nero Digital development - I can't say much now, but we are planning couple of big surprises that will be announced soon smile.gif

Once again, thank you all for the big support!

This post has been edited by Ivan Dimkovic: Dec 10 2004, 13:19
Go to the top of the page
+Quote Post
guruboolez
post Dec 10 2004, 13:52
Post #18





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



Hello Ivan,
first: BRAAAVOOO! You (and other people) did an impressing job with Nero AAC. Honestly, I didn't think that such progress could be possible in less than one year. And thanks for the explanation.

second: here is a sample, probably the worse I heard for Nero VBR 'fast' internet profile with regular music. The artifact is very common (noise is missing and fluctuant), and occurs mainly on the second part of the sample. The problem is audible on regular listening volume. I've found some samples showing this behaviour, but this one is the worse I've heard.

P.S. There's also (small but real) problems with pre-echo. But you probably know this.

This post has been edited by guruboolez: Dec 10 2004, 13:53
Go to the top of the page
+Quote Post
guruboolez
post Dec 10 2004, 14:17
Post #19





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (QuantumKnot @ Dec 9 2004, 01:18 PM)
Just as a sidenote, I found some free-time this week playing with some wavelet-based transient analysis (for block switching and dynamic impulse adjustment) for Vorbis 1.1.  Nothing concrete to release yet, but the results look quite interesting.  Expect an alpha of QKTune 2 sometime in the new year.  cool.gif
*

If you need some help for tests or opinions about it, count on me (if I'm not too busy).
Go to the top of the page
+Quote Post
QuantumKnot
post Dec 11 2004, 01:15
Post #20





Group: Developer
Posts: 1245
Joined: 16-December 02
From: Australia
Member No.: 4097



QUOTE (guruboolez @ Dec 10 2004, 11:17 PM)
If you need some help for tests or opinions about it, count on me (if I'm not too busy).
*


Many thanks. smile.gif
Go to the top of the page
+Quote Post
alter4
post Dec 13 2004, 10:54
Post #21





Group: Members
Posts: 110
Joined: 14-September 04
From: Belarus, Vitebsk
Member No.: 16992



I made some ABX test with guruboolez's samples compressed ~128 bitrate and...
for my ears NERO AAC not better than Aotuv beta3 аnd can even worse??
tested on Avison, Mozart - sonate and Bruhns - prelude.

Avison AAC 8/10
Avison Ao Beta3 7/10

Mozart AAC 7/10
Mozart Ao Beta3 7/10

Bruhns AAС 6/10
Bruhns Ao Beta3 7/10

I thank guruboolez for interesting sample.
blink.gif

This post has been edited by alter4: Dec 13 2004, 11:00
Go to the top of the page
+Quote Post
guruboolez
post Dec 13 2004, 10:57
Post #22





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



Strange results for Bruhns.wav: it's one of the most wounded IMO by aoTuV.
Anyway, thanks for additionnal tests smile.gif

This post has been edited by guruboolez: Dec 13 2004, 10:59
Go to the top of the page
+Quote Post
alter4
post Dec 13 2004, 11:03
Post #23





Group: Members
Posts: 110
Joined: 14-September 04
From: Belarus, Vitebsk
Member No.: 16992



QUOTE
Strange results for Bruhns.wav: it's one of the most wounded IMO by aoTuV.

sorry, typing error
I have corrected

This post has been edited by alter4: Dec 13 2004, 11:29
Go to the top of the page
+Quote Post
guruboolez
post Dec 13 2004, 11:35
Post #24





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



I've just noticed:
results below 8/10 are not significant.
8/10 -> pval = 0.055
7/10 -< pval = 0.172

http://ff123.net/abx/abx.php
Go to the top of the page
+Quote Post
ff123
post Dec 14 2004, 02:39
Post #25


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



QUOTE (guruboolez @ Dec 13 2004, 02:35 AM)
I've just noticed:
results below 8/10 are not significant.
8/10 -> pval = 0.055
7/10 -< pval = 0.172

http://ff123.net/abx/abx.php
*


Yeah, I would only choose 8 trials if you're reasonably sure that you'll be able to detect a difference.

ff123
Go to the top of the page
+Quote Post

2 Pages V   1 2 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 31st October 2014 - 09:34