IPB

Welcome Guest ( Log In | Register )

Multiformat listening test @ ~64kbps: Results, Results and post-test discussion
IgorC
post Apr 12 2011, 00:40
Post #1





Group: Members
Posts: 1577
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



The test is finished, results are available here:

http://listening-tests.hydrogenaudio.org/igorc/results.html

Summary: CELT/Opus won, Apple HE-AAC is better than Nero HE-AAC, and Vorbis has caught up with Nero HE-AAC.
Go to the top of the page
+Quote Post
 
Start new topic
Replies
googlebot
post Apr 12 2011, 08:06
Post #2





Group: Members
Posts: 698
Joined: 6-March 10
Member No.: 78779



I'm stunned by the CELT/Opus results! I would have assumed that your toolbox is smaller than usually when you are targeting low-delay. And now Celt even beats the others by lengths.

Thanks for the great work, guys!
Go to the top of the page
+Quote Post
NullC
post Apr 14 2011, 05:00
Post #3





Group: Developer
Posts: 200
Joined: 8-July 03
Member No.: 7653



QUOTE (googlebot @ Apr 11 2011, 23:06) *
I'm stunned by the CELT/Opus results! I would have assumed that your toolbox is smaller than usually when you are targeting low-delay. And now Celt even beats the others by lengths.
Thanks for the great work, guys!


We were surprised when we started getting competitive with high latency codecs too, but we've had a little while to get used to that. HE-AAC was a bit more surprising, especially since so many things were going against us (the immaturity of our encoder, it's inadequacy for this application (VBR), etc.)

Low-latency work implies some serious compromises, but it's not all bad. Small transform sizes automatically give you better time domain noise shaping, for example. There have been a lot of people that liked codecs like MPC at high rates for the reduced time-domain-liability they imply. Simply getting many little details right helps reduce the harm of low-latency. E.g. we use a trivial time domain pre-whitening before the transform so that the quantization noise from spectral leakage is less exposed.

We also expanded the low latency toolbox some. For example, we have short and long frames without window switching (and the look-ahead latency that requires). We're also using other things like range coding and high-dimensional vector quantization which might not have been great options a number of years ago. Our decoder is currently quite a bit slower than AAC decoders (though it's not optimized, so it's hard to say how much the ultimate difference will be), but since we're mostly targeting interactive use we were able to "pay" for decoder complexity increases with encoder decreases: We're an order of magnitude faster encode side than our high latency competition (no explicit psy-model!). With some fairly modest compromises you can make an Opus audio mode (the CELT part) encoder which is basically the same speed as the decoder. Though cpu cycle hungry, Opus uses a very small amount of memory, which eliminates one of the embedded hurdles Vorbis suffered.

I also like to think that the ultra-low latency support has fostered some beneficial discipline: Every single fractional-bit of signaling overhead and frame redundancy counts a lot with 2.5ms frames. While it's not so important with larger frames, waste is waste, and Opus has very little of it. A very high percentage of the bits in opus frames go directly to improving the audio band SNR, and very few go to shuffling around coding modes or switching between inconsequentially different alternatives. Signaling bits are sometimes helpful, sometimes _very_ helpful... but bits spent coding signal data are pretty much always helpful. We came up with a number of clever ways of eliminating signaling, and Opus is able to provide a true hard CBR (no bitres!) in audio modes which is super efficient (uses every bit in the frame) and actually sounds really good.

For music at lower rates, I expect that HE-AAC would winó we simply start to fall apart once the number of bytes in the frame gets too small. Speech is another matter and Opus should do quite well there down to very low rates, owing to the merger of Skype's SILK.

Opus should also scale well to higher ratesó it is not using any highly parametric techniques that don't respond to additional bitsó though the lack of a mature encoder will probably still give other codecs the edge in many cases. This is especially true for exposed multi-tonal samples like the sample 02 in this set, though multi-tonal killers are fairly uncommon and I expect that we can fix them with VBR in the same way large block codecs fix transients...

I also think that aggressive tuning of these HE-AAC encoders could put them back in the lead, or at least strongly tie opus. E.g. From my own listening I think a lot of the difference between nero and apple in the test was due to the lowpass difference for example. That said, the HE-AAC format is mature (and by some accounts now stagnant), and we have a lot of low hanging fruit.

I feel that Vorbis loses for a sad reason at this rate and lower: The Vorbis toolbox doesn't have any good tools to avoid having to low-pass at rates well below its initial design goals at higher rates it obviously does much better than it did here. The lack of efficient tools for low rate HF is real shortcoming at this rate, but not one which is all that interesting from an engineering/competitive basis.

Cheers,

This post has been edited by NullC: Apr 14 2011, 05:04
Go to the top of the page
+Quote Post
Garf
post Apr 14 2011, 12:19
Post #4


Server Admin


Group: Admin
Posts: 4886
Joined: 24-September 01
Member No.: 13



QUOTE (NullC @ Apr 14 2011, 06:00) *
Low-latency work implies some serious compromises, but it's not all bad. Small transform sizes automatically give you better time domain noise shaping, for example.


But not enough, given the presence of block-switching in Opus smile.gif

I could imagine the more frequent transmission of band energies is very useful. It should also help the folding. IIRC, SBR has issues with not being able to adapt fast enough time-domain-wise in some circumstances.

QUOTE
I also like to think that the ultra-low latency support has fostered some beneficial discipline: Every single fractional-bit of signaling overhead and frame redundancy counts a lot with 2.5ms frames. While it's not so important with larger frames, waste is waste, and Opus has very little of it. A very high percentage of the bits in opus frames go directly to improving the audio band SNR, and very few go to shuffling around coding modes or switching between inconsequentially different alternatives. Signaling bits are sometimes helpful, sometimes _very_ helpful... but bits spent coding signal data are pretty much always helpful. We came up with a number of clever ways of eliminating signaling, and Opus is able to provide a true hard CBR (no bitres!) in audio modes which is super efficient (uses every bit in the frame) and actually sounds really good.


I think you have a good advantage over (LC)AAC here: AAC allows very fine control of the quantization, but at a severe signaling cost. At least in AAC codec design, you can spend a very long time on the complicated question of doing the joint R/D optimization quickly, and end up not using the fine control at all because it just eats too many bits. This gets worse at low bitrates. It doesn't help AAC only uses huffman coding, and not arithmetic/range coding. H.264 also has many possible modes but I presume CABAC helps mitigate the signaling cost (maybe someone who is more familiar with that particular codec can confirm/deny).

Is the almost complete lack of signaling in Opus related to the decision not to make range coding contexts? If I read the spec correctly, almost all of the range coding assumes uniform distribution.

The 3GPP reference code (which is actually Fraunhofer fastaac, as far as I know) shows that you can make a decent AAC codec even ignoring most of the psycho-acoustics or greatly simplifying them, so I'm not surprised you eliminated the explicit psymodel entirely. It's a surprisingly small part of the codecs efficiency (I'm not saying its not important - it is - but less as you would think at first). VBR is another matter, though it's also surprisingly hard to make consistently good decisions there.

QUOTE
For music at lower rates, I expect that HE-AAC would winó we simply start to fall apart once the number of bytes in the frame gets too small.


This is a bit surprising to me because of the above. Are there technical limitations that cause this?

QUOTE
This is especially true for exposed multi-tonal samples like the sample 02 in this set, though multi-tonal killers are fairly uncommon and I expect that we can fix them with VBR in the same way large block codecs fix transients...


Did the codec used in this test use the tonal pre/postfilter from Broadcom?

QUOTE
I feel that Vorbis loses for a sad reason at this rate and lower: The Vorbis toolbox doesn't have any good tools to avoid having to low-pass at rates well below its initial design goals at higher rates it obviously does much better than it did here. The lack of efficient tools for low rate HF is real shortcoming at this rate, but not one which is all that interesting from an engineering/competitive basis.


I understand this as saying that Vorbis would be easily competitive if it had something like SBR or folding. The experience with LC-AAC vs HE-AAC seems to support that.
Go to the top of the page
+Quote Post

Posts in this topic
- IgorC   Multiformat listening test @ ~64kbps: Results   Apr 12 2011, 00:40
- - Garf   If someone can assist with a bitrate table or per-...   Apr 12 2011, 01:02
- - Garf   Oh, and given that Opus is open sourced, if one of...   Apr 12 2011, 01:06
- - AllanP   I just wonder one thing, when the Vorbis encoder w...   Apr 12 2011, 01:14
|- - Garf   QUOTE (AllanP @ Apr 12 2011, 02:14) I jus...   Apr 12 2011, 01:15
|- - AllanP   QUOTE (Garf @ Apr 12 2011, 02:15) You can...   Apr 12 2011, 01:22
- - romor   Congratulation to CELT/Opus! I wanted to com...   Apr 12 2011, 03:08
- - IgorC   I think the results of lessthanjoey and AlexB are ...   Apr 12 2011, 03:50
- - googlebot   I'm stunned by the CELT/Opus results! I wo...   Apr 12 2011, 08:06
|- - NullC   QUOTE (googlebot @ Apr 11 2011, 23:06) I...   Apr 14 2011, 05:00
|- - saratoga   QUOTE (NullC @ Apr 14 2011, 00:00) We als...   Apr 14 2011, 06:29
||- - NullC   QUOTE (saratoga @ Apr 13 2011, 22:29) Is ...   Apr 14 2011, 08:30
|- - Garf   QUOTE (NullC @ Apr 14 2011, 06:00) Low-la...   Apr 14 2011, 12:19
|- - jmvalin   QUOTE (Garf @ Apr 14 2011, 07:19) Is the ...   Apr 14 2011, 14:04
|- - NullC   QUOTE (Garf @ Apr 14 2011, 03:19) QUOTE (...   Apr 14 2011, 17:47
|- - C.R.Helmrich   QUOTE (NullC @ Apr 14 2011, 18:47) The sw...   Apr 14 2011, 23:39
|- - jmvalin   QUOTE (C.R.Helmrich @ Apr 14 2011, 18:39)...   Apr 15 2011, 05:49
- - Alex B   Thanks guys! Interesting results. One note t...   Apr 12 2011, 12:59
|- - Garf   QUOTE (Alex B @ Apr 12 2011, 13:59) I got...   Apr 12 2011, 13:53
|- - NullC   QUOTE For processing the result .txt files with ch...   Apr 12 2011, 14:48
|- - Garf   QUOTE (NullC @ Apr 12 2011, 15:48) Sounds...   Apr 12 2011, 14:59
||- - Alex B   QUOTE (Garf @ Apr 12 2011, 16:59) But the...   Apr 12 2011, 15:09
|- - Alex B   QUOTE (NullC @ Apr 12 2011, 16:48) Sounds...   Apr 12 2011, 15:02
|- - NullC   QUOTE (Alex B @ Apr 12 2011, 07:02) QUOTE...   Apr 12 2011, 15:17
|- - motion_blur   QUOTE (Alex B @ Apr 12 2011, 16:02) QUOTE...   Apr 12 2011, 16:15
|- - NullC   QUOTE (motion_blur @ Apr 12 2011, 08:15) ...   Apr 12 2011, 17:54
|- - motion_blur   QUOTE (NullC @ Apr 12 2011, 18:54) QUOTE ...   Apr 12 2011, 19:42
- - Alex B   For comparison I uploaded a rar package of my ...   Apr 12 2011, 14:14
|- - Garf   QUOTE (Alex B @ Apr 12 2011, 15:14) For c...   Apr 12 2011, 14:49
- - Alex B   QUOTE (Garf @ Apr 12 2011, 16:59) But the...   Apr 12 2011, 15:35
|- - Garf   QUOTE (Alex B @ Apr 12 2011, 16:35) QUOTE...   Apr 12 2011, 15:42
- - Alex B   Regarding the bitrate table, I guess that CELT/Op...   Apr 12 2011, 16:14
|- - NullC   QUOTE (Alex B @ Apr 12 2011, 08:14) Regar...   Apr 12 2011, 18:10
- - IgorC   Yes, I was too strict. Sorry about it. Some of th...   Apr 12 2011, 18:13
|- - motion_blur   QUOTE (IgorC @ Apr 12 2011, 19:13) Yes, I...   Apr 12 2011, 20:09
|- - NullC   QUOTE (motion_blur @ Apr 12 2011, 12:09) ...   Apr 13 2011, 00:59
|- - motion_blur   QUOTE (NullC @ Apr 13 2011, 01:59) QUOTE ...   Apr 13 2011, 10:06
- - markanini   I figured ratings would vary between testers depen...   Apr 12 2011, 18:52
|- - NullC   QUOTE (markanini @ Apr 12 2011, 09:52) I ...   Apr 12 2011, 20:57
- - lessthanjoey   I've done some more testing with headphones af...   Apr 12 2011, 19:51
- - C.R.Helmrich   Thanks for organizing the tests, guys! Sorry f...   Apr 12 2011, 20:11
|- - motion_blur   QUOTE (C.R.Helmrich @ Apr 12 2011, 21:11)...   Apr 12 2011, 20:41
|- - Garf   QUOTE Please provide the number of valid results (...   Apr 12 2011, 21:44
||- - C.R.Helmrich   Sorry, Christoph, can't reproduce it. What you...   Apr 12 2011, 22:12
||- - Garf   QUOTE (C.R.Helmrich @ Apr 12 2011, 23:12)...   Apr 12 2011, 23:37
||- - motion_blur   QUOTE (C.R.Helmrich @ Apr 12 2011, 23:12)...   Apr 13 2011, 00:41
|- - NullC   QUOTE (C.R.Helmrich @ Apr 12 2011, 11:11)...   Apr 12 2011, 21:47
- - IgorC   motion_blur, You can download the results of all...   Apr 12 2011, 20:21
|- - _m≤_   Some presentation suggestions: 1. Codec versions a...   Apr 12 2011, 20:43
|- - IgorC   QUOTE (_m≤_ @ Apr 12 2011, 16:43) Some pr...   Apr 12 2011, 20:46
|- - Alex B   QUOTE (_m≤_ @ Apr 12 2011, 22:43) 2. Link...   Apr 12 2011, 21:06
||- - Alex B   QUOTE (Alex B @ Apr 12 2011, 23:06) QUOTE...   Apr 13 2011, 11:38
||- - NullC   QUOTE (Alex B @ Apr 13 2011, 02:38) The b...   Apr 13 2011, 12:41
||- - Garf   QUOTE (NullC @ Apr 13 2011, 13:41) QUOTE ...   Apr 13 2011, 12:54
||- - Alex B   QUOTE (NullC @ Apr 13 2011, 14:41) Any id...   Apr 13 2011, 14:46
||- - NullC   QUOTE (Alex B @ Apr 13 2011, 05:46) QUOTE...   Apr 13 2011, 22:48
|- - Garf   QUOTE (_m≤_ @ Apr 12 2011, 21:43) Some pr...   Apr 12 2011, 22:24
- - Alex B   Here is the raw data for a bitrate table. The bitr...   Apr 12 2011, 20:30
- - IgorC   Thank you for your help, AlexB. If you can do the ...   Apr 12 2011, 20:44
- - C.R.Helmrich   Christoph, do you mean the slightly washed out bas...   Apr 12 2011, 20:52
|- - motion_blur   QUOTE (C.R.Helmrich @ Apr 12 2011, 21:52)...   Apr 12 2011, 21:11
|- - C.R.Helmrich   QUOTE (motion_blur @ Apr 12 2011, 22:11) ...   Apr 12 2011, 21:38
|- - motion_blur   QUOTE (C.R.Helmrich @ Apr 12 2011, 22:38)...   Apr 12 2011, 21:56
- - IgorC   I've checked. The decoder on Christoph's s...   Apr 12 2011, 20:54
- - Alex B   Here's the bitrate table: In Excel format: ...   Apr 12 2011, 22:20
|- - saintdev   QUOTE (Alex B @ Apr 12 2011, 14:20) Here...   Apr 12 2011, 23:21
- - NullC   QUOTE (IgorC @ Apr 11 2011, 16:40) The te...   Apr 13 2011, 04:42
|- - Garf   QUOTE (NullC @ Apr 13 2011, 05:42) Hey al...   Apr 13 2011, 08:27
||- - IgorC   QUOTE (Garf @ Apr 13 2011, 04:27) One thi...   Apr 14 2011, 05:57
|- - C.R.Helmrich   Thanks, Garf, for the plot! And thanks, Christ...   Apr 13 2011, 22:46
|- - Garf   QUOTE (C.R.Helmrich @ Apr 13 2011, 23:46)...   Apr 13 2011, 22:57
||- - C.R.Helmrich   QUOTE (Garf @ Apr 13 2011, 23:57) From th...   Apr 13 2011, 23:06
|- - jmvalin   QUOTE (C.R.Helmrich @ Apr 13 2011, 17:46)...   Apr 14 2011, 00:58
|- - Garf   QUOTE (jmvalin @ Apr 14 2011, 01:58) I do...   Apr 14 2011, 09:13
|- - jmvalin   QUOTE (Garf @ Apr 14 2011, 04:13) You are...   Apr 14 2011, 11:41
- - Garf   The result page is now updated with per-sample gra...   Apr 13 2011, 12:41
- - mixminus1   Thanks much to all for their work in both setting ...   Apr 13 2011, 14:55
|- - Garf   QUOTE (mixminus1 @ Apr 13 2011, 15:55) Th...   Apr 13 2011, 15:04
- - mixminus1   :facepalm: Good God... Thanks, Garf, I was scour...   Apr 13 2011, 15:15
- - romor   @Garf: can you please reupload results you posted ...   Apr 13 2011, 16:35
|- - Garf   QUOTE (romor @ Apr 13 2011, 17:35) @Garf:...   Apr 13 2011, 18:59
- - IgorC   AlexB, thank you for bitrate verification. I real...   Apr 13 2011, 17:12
- - pdq   Whether or not classical should be considered to b...   Apr 13 2011, 17:51
|- - IgorC   QUOTE (pdq @ Apr 13 2011, 13:51) Whether ...   Apr 13 2011, 18:01
|- - pdq   QUOTE (IgorC @ Apr 13 2011, 13:01) QUOTE ...   Apr 13 2011, 18:53
- - romor   file: http://people.xiph.org/~greg/opus/ha2011/2.....   Apr 13 2011, 19:28
|- - NullC   QUOTE (romor @ Apr 13 2011, 10:28) file: ...   Apr 13 2011, 19:53
- - IgorC   Bitrate verification on my set of albums: http:/...   Apr 13 2011, 21:45
|- - NullC   QUOTE (IgorC @ Apr 13 2011, 13:45) Bitrat...   Apr 21 2011, 19:15
|- - jmvalin   QUOTE (NullC @ Apr 21 2011, 14:15) QUOTE ...   Apr 21 2011, 19:53
- - Garf   QUOTE Has anyone ever seriously blind-tested e.g. ...   Apr 14 2011, 00:10
|- - C.R.Helmrich   QUOTE (Garf @ Apr 14 2011, 01:10) I'm...   Apr 14 2011, 10:46
- - .alexander.   The second graph seems to be consistent with ...   Apr 15 2011, 12:40
|- - jmvalin   QUOTE (.alexander. @ Apr 15 2011, 07:40) ...   Apr 15 2011, 15:17
- - Xanikseo   QUOTE (IgorC @ Apr 13 2011, 21:45) Bitrat...   Apr 20 2011, 16:34
|- - Garf   QUOTE (Xanikseo @ Apr 20 2011, 17:34) Igo...   Apr 20 2011, 18:02
|- - IgorC   QUOTE (Xanikseo @ Apr 20 2011, 12:34) Igo...   Apr 20 2011, 18:19
|- - Zarggg   QUOTE (Xanikseo @ Apr 20 2011, 11:34) EDI...   Apr 20 2011, 19:09
- - Xanikseo   QUOTE (Zarggg @ Apr 20 2011, 19:09) QUOTE...   Apr 20 2011, 21:20
- - IgorC   NullC, h*tp://www.mediafire.com/?s7i9usu2qr27pcg ...   Apr 21 2011, 23:52
2 Pages V   1 2 >


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 20th October 2014 - 11:18