IPB

Welcome Guest ( Log In | Register )

Personal Listening Test of Opus, Celt, AAC at 75-100kbps, ABC/HR blind test, 1 Listener
Kamedo2
post Nov 17 2012, 09:25
Post #1





Group: Members
Posts: 181
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



Abstract:
Blind Comparison between 2012/09 new Opusenc(tfsel5), old Celtenc 0.11.2, Apple AAC-LC tvbr, cvbr.
This is an English version of my original post in Japanese. http://d.hatena.ne.jp/kamedo2/20121116/1353099244#seemore

Encoders:
libopus 0.9.11-146-gdc4f83b-exp_analysis
https://people.xiph.org/~greg/opus-tools_exp_dc4f83be.zip
celt-0.11.2-win32
https://people.xiph.org/~greg/celt-0.11.2-win32.zip
qaac 1.40
qaac 1.40

Settings:
opusenc --bitrate 66 input.wav output.wav
celtenc input.48k.raw --bitrate 75 --comp 10 output.wav
qaac --cvbr 72 -o output.m4a input.wav
qaac --tvbr 27 -o output.m4a input.wav
opusenc --bitrate 90 input.wav output.wav
celtenc input.48k.raw --bitrate 100 --comp 10 output.wav
qaac --cvbr 96 -o output.m4a input.wav
qaac --tvbr 45 -o output.m4a input.wav

Samples:
20 Sounds of various genres, from easy to modestly critical.
http://zak.s206.xrea.com/bitratetest/main.htm
To download, access to the link above, 2nd paragraph, 3rd-6th links. (40_30sec - Run up)

Hardwares:
Sony PSP-3000 + RP-HT560(1st) , RP-HJE150(2nd), took the average of the two results.

Results:




Conclusions & Observations:
I could not detect a significant improvement in the new September 1st version of Opus, from the old Celtenc in 2011.
It's possibly because the new Opus inflates bitrates more than it improves qualities, although the set of sounds contain easy samples.
On 75kbps, Opus/Celt are markedly better. On 100kbps, there is no big difference between those codecs.

Raw data:
40 Logs and encoders, decoders log
http://zak.s206.xrea.com/bitratetest/log_o...kbps100kbps.zip
CODE
% Opus, AAC 75kbps, 100kbps ABC/HR Score
% This format is compatible with my graphmaker, as well as ff123's FRIEDMAN.
opus_75k    celt_75k    cvbr_75k    tvbr_75k    opus100k    celt100k    cvbr100k    tvbr100k
%features 6 75kbps 75kbps 75kbps 75kbps 100kbps 100kbps 100kbps 100kbps
%features 7 OPUS OPUS AAC-LC AAC-LC OPUS OPUS AAC-LC AAC-LC
3.050    3.100    2.500    2.750    3.500    3.750    3.700    3.800    
3.750    2.950    2.700    2.750    4.050    3.800    4.000    3.950    
2.800    2.550    3.000    3.000    3.600    3.250    4.050    3.900    
2.700    3.150    2.350    2.300    3.350    3.800    3.600    3.700    
4.000    3.400    2.850    2.850    4.350    3.900    3.550    3.550    
2.600    2.550    2.800    2.800    3.350    3.150    3.950    3.900    
3.400    3.950    3.000    3.200    3.850    4.500    3.700    3.800    
3.450    3.500    2.900    2.800    3.850    4.050    4.050    4.150    
2.950    2.700    3.550    3.450    3.250    3.450    4.000    3.850    
3.100    3.400    2.750    2.600    3.800    3.850    4.150    4.000    
3.350    3.100    2.600    2.600    3.750    3.400    3.450    3.500    
3.750    3.350    2.800    2.950    4.050    3.750    3.800    3.850    
3.550    3.300    2.600    2.650    4.250    3.950    3.750    3.600    
3.100    3.350    2.750    2.550    3.650    3.700    3.850    3.800    
3.400    3.450    2.900    2.900    3.650    3.950    3.750    3.900    
3.250    3.300    2.750    2.800    3.650    3.850    3.950    3.750    
3.600    3.800    3.300    3.300    3.550    4.000    3.650    3.700    
3.700    3.350    3.300    3.300    3.900    3.650    4.100    4.000    
3.100    3.600    3.150    3.000    3.700    3.800    4.100    3.850    
3.650    4.050    3.000    2.900    4.050    4.250    3.750    3.550

It's not strange that some scores get 0.050 scale because I did tests twice per each music.
Go to the top of the page
+Quote Post
 
Start new topic
Replies
Kamedo2
post Nov 21 2012, 02:08
Post #2





Group: Members
Posts: 181
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



QUOTE (jmvalin @ Nov 20 2012, 13:13) *
CELT would normally stay where it is because it never had unconstrained VBR. It was always trying to hit the requested bit-rate average for every file. Unconstrained VBR is a new feature in Opus and is still only implemented in the development branch (git master). So normally the CELT curves would remain in place, while the OPus curves would move to 66 kb/s and 90 kb/s.

I assume that the preference for bitrate 'calibration' is to make it close to the real-world applications, where easy and difficult samples appear roughly 1:1, as opposed to 2:18 now.
In the real world where easy samples appear more frequently than in this test, cbr codecs like CELT will not 'normally stay'. Because there are plenty of easy samples, it means less perceptible artifacts, thus better average quality(Imagine MP3 CBR 128kbps), and move upwards. It will NOT remain in place.
In other words, in the typical music storage applications, VBRs like Opus get better compression, while CBRs get better quality.


QUOTE (C.R.Helmrich @ Nov 21 2012, 07:01) *
Edit: I put these samples into Fraunhofer's latest AAC encoder, and at VBR 3 (target ~97 kbps), they average 107 kbps. Here as well, only 2 items get <= 97 kbps (samples 11ff and 24td). So it seems that what Opus/Celt finds difficult to code is not that different from what an AAC encoder might find difficult, after all (except maybe for very tonal items, yes).

Chris

Interesting stuff. 11ff = finalfantasy is the samples Opus considered it to be the hardest. But for AAC, it's easy. I guess it's because the sound of the harpsichord is tonal. After using 101 and 136kbps for the harpsichord, Opus did a very good job preserving the detail(3.75 and 4.05, respectably). And somehow, 24td = Tom's Diner is easy for AAC(but difficult for Opus).
Go to the top of the page
+Quote Post
jmvalin
post Nov 21 2012, 03:15
Post #3


Xiph.org Speex developer


Group: Developer
Posts: 475
Joined: 21-August 02
Member No.: 3134



QUOTE (Kamedo2 @ Nov 20 2012, 20:08) *
I assume that the preference for bitrate 'calibration' is to make it close to the real-world applications, where easy and difficult samples appear roughly 1:1, as opposed to 2:18 now.
In the real world where easy samples appear more frequently than in this test, cbr codecs like CELT will not 'normally stay'. Because there are plenty of easy samples, it means less perceptible artifacts, thus better average quality(Imagine MP3 CBR 128kbps), and move upwards. It will NOT remain in place.
In other words, in the typical music storage applications, VBRs like Opus get better compression, while CBRs get better quality.


Actually, in the "real world", I would say that difficult samples represent less than 10% of all samples. Also, by "stay where it is", all I meant was that the x axis would remain at the same place regardless of the samples because CELT gives the same average for every sample (unlike the new Opus code). As for "better compression vs better quality", it's really all the same thing because you trade one for the other. The main thing we've actually been trying to do with VBR is to keep the quality constant and by adjusting the bitrate. In a perfect world (unachievable), all samples would get exactly the same quality and only the bit-rate would vary.


QUOTE (Kamedo2 @ Nov 20 2012, 20:08) *
Interesting stuff. 11ff = finalfantasy is the samples Opus considered it to be the hardest. But for AAC, it's easy. I guess it's because the sound of the harpsichord is tonal. After using 101 and 136kbps for the harpsichord, Opus did a very good job preserving the detail(3.75 and 4.05, respectably). And somehow, 24td = Tom's Diner is easy for AAC(but difficult for Opus).


Of all the instruments, harpsichord is the most difficult to code for Opus and the one that requires the highest bitrate. So the fact that you included two harpsichord samples (although harpsichord is typically rare) is in part what explains why Opus is using a rate that's much higher than it's usual rate. As for the reason why tonal samples are harder for Opus than for AAC, the main reason is the focus on interactive applications. Opus is designed for lower delay than AAC-LC, which forced the use of a shorter window (especially shorter overlap).
Go to the top of the page
+Quote Post
Kamedo2
post Nov 21 2012, 05:37
Post #4





Group: Members
Posts: 181
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



QUOTE (jmvalin @ Nov 21 2012, 11:15) *
Actually, in the "real world", I would say that difficult samples represent less than 10% of all samples. Also, by "stay where it is", all I meant was that the x axis would remain at the same place regardless of the samples because CELT gives the same average for every sample (unlike the new Opus code). As for "better compression vs better quality", it's really all the same thing because you trade one for the other. The main thing we've actually been trying to do with VBR is to keep the quality constant and by adjusting the bitrate. In a perfect world (unachievable), all samples would get exactly the same quality and only the bit-rate would vary.

I meant 'hard samples' by the definition of its real bitrate exceeds the value of the --bitrate option. In a real world, if we were to randomly choose 100 samples from the world, it is likely that roughly 50 samples < value of the --bitrate option < 50 samples. Yes, 1:1. As for "better compression vs better quality", those who want to calibrate the x-axis for the better representation of the world should also calibrate the y-axis. The calibration of the y-axis involves listening tests including many easy samples, and it's understandable why people don't want to do that, but on the average bitrate vs average score graph, calibrating only one axis would break the beautiful point of the graph, when the graph includes evaluation of CBR codecs like CELT, and make it less fair.

By far the simplest solution I think of is to add more easy samples, as defined by samples that creates less than expected from the value of the --bitrate option.

QUOTE (jmvalin @ Nov 21 2012, 11:15) *
Of all the instruments, harpsichord is the most difficult to code for Opus and the one that requires the highest bitrate. So the fact that you included two harpsichord samples (although harpsichord is typically rare) is in part what explains why Opus is using a rate that's much higher than it's usual rate.

Sorry for my misleading reply. Out of 20 samples I used, only finalfantasy is a harpsichord sample. I can understand why would critical samples like finalfantasy, FloorEssence, or VelvetRealm takes much space.
Go to the top of the page
+Quote Post

Posts in this topic
- Kamedo2   Personal Listening Test of Opus, Celt, AAC at 75-100kbps   Nov 17 2012, 09:25
- - C.R.Helmrich   Thanks for this interesting test, Kamedo, and welc...   Nov 17 2012, 10:28
|- - Kamedo2   QUOTE (C.R.Helmrich @ Nov 17 2012, 18:28)...   Nov 17 2012, 10:40
- - Dynamic   Thank you for your time and dedication, Kamedo2 ...   Nov 17 2012, 10:48
- - IgorC   Kamedo2, Thank You for all your tests. Glad to see...   Nov 17 2012, 11:04
- - Kamedo2   The samples I used The ABX criteria is 12/15(p=0...   Nov 17 2012, 18:44
- - Anakunda   QUOTE (Kamedo2 @ Nov 17 2012, 09:25) Blin...   Nov 17 2012, 23:45
|- - C.R.Helmrich   QUOTE (Anakunda @ Nov 18 2012, 00:45) Is ...   Nov 18 2012, 00:54
|- - Kamedo2   QUOTE (Anakunda @ Nov 18 2012, 07:45) Is ...   Nov 18 2012, 05:47
- - Dynamic   QUOTE x-axis=actual bitrate That was one query I ...   Nov 18 2012, 09:20
|- - Kamedo2   QUOTE (Dynamic @ Nov 18 2012, 17:20) QUOT...   Nov 18 2012, 10:23
- - Dynamic   Thank you for the clarification. It seems that du...   Nov 18 2012, 20:07
- - IgorC   Kamedo2, I'm not here to criticize your test...   Nov 18 2012, 20:12
|- - Kamedo2   QUOTE (IgorC @ Nov 19 2012, 04:12) As all...   Nov 19 2012, 02:17
|- - jmvalin   Hi Kamedo2, thanks for the test. From what I see, ...   Nov 19 2012, 20:03
- - lvqcl   I took my Opus compile (libopus v1.0.1-140-gc55f4d...   Nov 19 2012, 20:32
|- - Kamedo2   QUOTE (jmvalin @ Nov 20 2012, 04:03) it w...   Nov 19 2012, 23:19
|- - jmvalin   QUOTE (Kamedo2 @ Nov 19 2012, 17:19) Acco...   Nov 20 2012, 18:39
- - Dynamic   Thanks again to everyone in this thread. I'm c...   Nov 20 2012, 03:03
|- - jmvalin   QUOTE (Dynamic @ Nov 19 2012, 21:03) We d...   Nov 20 2012, 05:13
- - C.R.Helmrich   QUOTE (Kamedo2 @ Nov 17 2012, 10:25) Samp...   Nov 20 2012, 23:01
- - Kamedo2   QUOTE (jmvalin @ Nov 20 2012, 13:13) CELT...   Nov 21 2012, 02:08
|- - jmvalin   QUOTE (Kamedo2 @ Nov 20 2012, 20:08) I as...   Nov 21 2012, 03:15
|- - Kamedo2   QUOTE (jmvalin @ Nov 21 2012, 11:15) Actu...   Nov 21 2012, 05:37
|- - jmvalin   QUOTE (Kamedo2 @ Nov 20 2012, 23:37) I me...   Nov 22 2012, 02:45
|- - Kamedo2   QUOTE (jmvalin @ Nov 22 2012, 10:45) QUOT...   Nov 22 2012, 07:33
|- - jmvalin   QUOTE (Kamedo2 @ Nov 22 2012, 01:33) Ther...   Nov 22 2012, 18:10
- - Kamedo2   Bitrate vs Score plot of the 20 samples used. Opu...   Nov 21 2012, 03:26
- - Dynamic   I think the objectives in tests (experiments) matt...   Nov 22 2012, 19:03
|- - Kamedo2   QUOTE (Dynamic @ Nov 23 2012, 03:03) Ques...   Nov 22 2012, 22:42
- - Kamedo2   QUOTE rjamorim: There's some inverse proportio...   Nov 23 2012, 16:35
- - Kamedo2   I measured an average bitrate over wide range of n...   Nov 23 2012, 21:51
|- - jmvalin   QUOTE (Kamedo2 @ Nov 23 2012, 15:51) The ...   Nov 24 2012, 19:49
|- - Kamedo2   QUOTE (jmvalin @ Nov 25 2012, 03:49) QUOT...   Nov 25 2012, 15:34
- - Kamedo2   My post #34 might be too difficult. I wish I had b...   Nov 25 2012, 21:35
- - IgorC   Interesting. The Opus'es scores have less devi...   Nov 26 2012, 03:08
|- - Kamedo2   QUOTE (IgorC @ Nov 26 2012, 11:08) The Op...   Nov 26 2012, 23:56
|- - DonP   QUOTE (IgorC @ Nov 25 2012, 21:08) In thi...   Jan 3 2013, 02:57
- - Dynamic   Once again, Kamedo2, I applaud you for your testin...   Nov 26 2012, 21:57
|- - Kamedo2   QUOTE (Dynamic @ Nov 27 2012, 05:57) As I...   Nov 27 2012, 00:14
- - jmvalin   Kamedo2, can you give 1.1-alpha a try? It includes...   Jan 3 2013, 01:09
- - Kamedo2   QUOTE (jmvalin @ Jan 3 2013, 09:09) Kamed...   Jan 5 2013, 11:50


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 24th July 2014 - 05:59