IPB

Welcome Guest ( Log In | Register )

Personal Listening Test of Opus, Celt, AAC at 75-100kbps, ABC/HR blind test, 1 Listener
Kamedo2
post Nov 17 2012, 09:25
Post #1





Group: Members
Posts: 220
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



Abstract:
Blind Comparison between 2012/09 new Opusenc(tfsel5), old Celtenc 0.11.2, Apple AAC-LC tvbr, cvbr.
This is an English version of my original post in Japanese. http://d.hatena.ne.jp/kamedo2/20121116/1353099244#seemore

Encoders:
libopus 0.9.11-146-gdc4f83b-exp_analysis
https://people.xiph.org/~greg/opus-tools_exp_dc4f83be.zip
celt-0.11.2-win32
https://people.xiph.org/~greg/celt-0.11.2-win32.zip
qaac 1.40
qaac 1.40

Settings:
opusenc --bitrate 66 input.wav output.wav
celtenc input.48k.raw --bitrate 75 --comp 10 output.wav
qaac --cvbr 72 -o output.m4a input.wav
qaac --tvbr 27 -o output.m4a input.wav
opusenc --bitrate 90 input.wav output.wav
celtenc input.48k.raw --bitrate 100 --comp 10 output.wav
qaac --cvbr 96 -o output.m4a input.wav
qaac --tvbr 45 -o output.m4a input.wav

Samples:
20 Sounds of various genres, from easy to modestly critical.
http://zak.s206.xrea.com/bitratetest/main.htm
To download, access to the link above, 2nd paragraph, 3rd-6th links. (40_30sec - Run up)

Hardwares:
Sony PSP-3000 + RP-HT560(1st) , RP-HJE150(2nd), took the average of the two results.

Results:




Conclusions & Observations:
I could not detect a significant improvement in the new September 1st version of Opus, from the old Celtenc in 2011.
It's possibly because the new Opus inflates bitrates more than it improves qualities, although the set of sounds contain easy samples.
On 75kbps, Opus/Celt are markedly better. On 100kbps, there is no big difference between those codecs.

Raw data:
40 Logs and encoders, decoders log
http://zak.s206.xrea.com/bitratetest/log_o...kbps100kbps.zip
CODE
% Opus, AAC 75kbps, 100kbps ABC/HR Score
% This format is compatible with my graphmaker, as well as ff123's FRIEDMAN.
opus_75k    celt_75k    cvbr_75k    tvbr_75k    opus100k    celt100k    cvbr100k    tvbr100k
%features 6 75kbps 75kbps 75kbps 75kbps 100kbps 100kbps 100kbps 100kbps
%features 7 OPUS OPUS AAC-LC AAC-LC OPUS OPUS AAC-LC AAC-LC
3.050    3.100    2.500    2.750    3.500    3.750    3.700    3.800    
3.750    2.950    2.700    2.750    4.050    3.800    4.000    3.950    
2.800    2.550    3.000    3.000    3.600    3.250    4.050    3.900    
2.700    3.150    2.350    2.300    3.350    3.800    3.600    3.700    
4.000    3.400    2.850    2.850    4.350    3.900    3.550    3.550    
2.600    2.550    2.800    2.800    3.350    3.150    3.950    3.900    
3.400    3.950    3.000    3.200    3.850    4.500    3.700    3.800    
3.450    3.500    2.900    2.800    3.850    4.050    4.050    4.150    
2.950    2.700    3.550    3.450    3.250    3.450    4.000    3.850    
3.100    3.400    2.750    2.600    3.800    3.850    4.150    4.000    
3.350    3.100    2.600    2.600    3.750    3.400    3.450    3.500    
3.750    3.350    2.800    2.950    4.050    3.750    3.800    3.850    
3.550    3.300    2.600    2.650    4.250    3.950    3.750    3.600    
3.100    3.350    2.750    2.550    3.650    3.700    3.850    3.800    
3.400    3.450    2.900    2.900    3.650    3.950    3.750    3.900    
3.250    3.300    2.750    2.800    3.650    3.850    3.950    3.750    
3.600    3.800    3.300    3.300    3.550    4.000    3.650    3.700    
3.700    3.350    3.300    3.300    3.900    3.650    4.100    4.000    
3.100    3.600    3.150    3.000    3.700    3.800    4.100    3.850    
3.650    4.050    3.000    2.900    4.050    4.250    3.750    3.550

It's not strange that some scores get 0.050 scale because I did tests twice per each music.
Go to the top of the page
+Quote Post
 
Start new topic
Replies
IgorC
post Nov 18 2012, 20:12
Post #2





Group: Members
Posts: 1581
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Kamedo2,

I'm not here to criticize your test, everything in contrary. I'm agree with your results.
I know how it feels when somebody starts criticize results of test because someone doesn't like that the codec he includes in his products wasn't first and no matter how good your intentions were.

So, Thank You Very Much for your labor. smile.gif

Though I have a bit different philosophy of testing VBR encoders which are very closely related to HA's public tests one.
Simply talking, it's to expect that encoder bloats bitrate on difficult samples. It's ok while VBR encoders respect the average bitrate on a large number of albums.
But still I agree with You that it's not wrong to think that an overall bitrate for all tested samples should be the same too.
As all listening tests yours has samples which difficultness higher than average. Not necessary extremely killers but somewhat harder to code. So it's an evaluation of HIGH part of VBR only. But we don't test here simple parts where the encoder goes really LOW. Yes, the experimental Opus bloats bitrate on hard parts (HIGH) but there wasn't an evaluation of a LOW part (very easy to encode samples). More likely Opus'es VBR mode is more "true" than Apple's true VBR laugh.gif
I've encoded some albums with the experimental Opus build and CELT 0.11.2. Yes, there could be difference in 2-3 kbps overall but shifting bitrates ~10 kbps is a bit too much.

But then again it's my point of view.

I think ideally two encoders should be tested at bitrate where both end up with the same file size for large number of albums and then randomly choose testing items to match the same rate between them too.

This post has been edited by IgorC: Nov 18 2012, 20:50
Go to the top of the page
+Quote Post
Kamedo2
post Nov 19 2012, 02:17
Post #3





Group: Members
Posts: 220
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



QUOTE (IgorC @ Nov 19 2012, 04:12) *
As all listening tests yours has samples which difficultness higher than average. Not necessary extremely killers but somewhat harder to code. So it's an evaluation of HIGH part of VBR only. But we don't test here simple parts where the encoder goes really LOW. Yes, the experimental Opus bloats bitrate on hard parts (HIGH) but there wasn't an evaluation of a LOW part (very easy to encode samples).

If we were to include LOW samples, we can predict what happens on CBR and VBR codecs.
On CBR like CELT, because there are plenty of bits available, the average quality will go up. It will make upward shift on the x-axis=bitrate y-axis=quality plot.
On VBR like Opus, because the goal is to retain the same quality, the average bitrate will go down, while quality remains the same. Leftward shift on the plot.
On the bitrate vs quality graph, upward shift and leftward shift effectively means the same thing; better results.

QUOTE (IgorC @ Nov 19 2012, 04:12) *
I've encoded some albums with the experimental Opus build and CELT 0.11.2. Yes, there could be difference in 2-3 kbps overall but shifting bitrates ~10 kbps is a bit too much.

Nice to hear that bitrate increase is rather minor.
The problem is, the "some albums" is not redistributable thus not reproducible. May be we should have an objective method to measure "bitrate bloat".
On VBR development people are likely to tweak settings that cause bitrate increase. Rightward shift on the plot is a bad thing, and the tweak involves bitrate increase on only the
critical part, average bitrate bloat will be minimized, meaning very little rightward shift. If the tweaks has an generic bitrate increase, rightward shift is big, it means useless.
Generic bitrate increase plagues LAME, and I don't want Opus to follow the same way.

As for my samples being short, a Vorbis codebook is, for example, several kilobytes. Compared to 4-5 minutes samples, there are at best 2% more benefit that Opus has from being small.
It can be a problem when the intended application is a music storage. On ads and news videos, game sounds, wikimedia, tests at 20 seconds is about right.
Go to the top of the page
+Quote Post

Posts in this topic
- Kamedo2   Personal Listening Test of Opus, Celt, AAC at 75-100kbps   Nov 17 2012, 09:25
- - C.R.Helmrich   Thanks for this interesting test, Kamedo, and welc...   Nov 17 2012, 10:28
|- - Kamedo2   QUOTE (C.R.Helmrich @ Nov 17 2012, 18:28)...   Nov 17 2012, 10:40
- - Dynamic   Thank you for your time and dedication, Kamedo2 ...   Nov 17 2012, 10:48
- - IgorC   Kamedo2, Thank You for all your tests. Glad to see...   Nov 17 2012, 11:04
- - Kamedo2   The samples I used The ABX criteria is 12/15(p=0...   Nov 17 2012, 18:44
- - Anakunda   QUOTE (Kamedo2 @ Nov 17 2012, 09:25) Blin...   Nov 17 2012, 23:45
|- - C.R.Helmrich   QUOTE (Anakunda @ Nov 18 2012, 00:45) Is ...   Nov 18 2012, 00:54
|- - Kamedo2   QUOTE (Anakunda @ Nov 18 2012, 07:45) Is ...   Nov 18 2012, 05:47
- - Dynamic   QUOTE x-axis=actual bitrate That was one query I ...   Nov 18 2012, 09:20
|- - Kamedo2   QUOTE (Dynamic @ Nov 18 2012, 17:20) QUOT...   Nov 18 2012, 10:23
- - Dynamic   Thank you for the clarification. It seems that du...   Nov 18 2012, 20:07
- - IgorC   Kamedo2, I'm not here to criticize your test...   Nov 18 2012, 20:12
|- - Kamedo2   QUOTE (IgorC @ Nov 19 2012, 04:12) As all...   Nov 19 2012, 02:17
|- - jmvalin   Hi Kamedo2, thanks for the test. From what I see, ...   Nov 19 2012, 20:03
- - lvqcl   I took my Opus compile (libopus v1.0.1-140-gc55f4d...   Nov 19 2012, 20:32
|- - Kamedo2   QUOTE (jmvalin @ Nov 20 2012, 04:03) it w...   Nov 19 2012, 23:19
|- - jmvalin   QUOTE (Kamedo2 @ Nov 19 2012, 17:19) Acco...   Nov 20 2012, 18:39
- - Dynamic   Thanks again to everyone in this thread. I'm c...   Nov 20 2012, 03:03
|- - jmvalin   QUOTE (Dynamic @ Nov 19 2012, 21:03) We d...   Nov 20 2012, 05:13
- - C.R.Helmrich   QUOTE (Kamedo2 @ Nov 17 2012, 10:25) Samp...   Nov 20 2012, 23:01
- - Kamedo2   QUOTE (jmvalin @ Nov 20 2012, 13:13) CELT...   Nov 21 2012, 02:08
|- - jmvalin   QUOTE (Kamedo2 @ Nov 20 2012, 20:08) I as...   Nov 21 2012, 03:15
|- - Kamedo2   QUOTE (jmvalin @ Nov 21 2012, 11:15) Actu...   Nov 21 2012, 05:37
|- - jmvalin   QUOTE (Kamedo2 @ Nov 20 2012, 23:37) I me...   Nov 22 2012, 02:45
|- - Kamedo2   QUOTE (jmvalin @ Nov 22 2012, 10:45) QUOT...   Nov 22 2012, 07:33
|- - jmvalin   QUOTE (Kamedo2 @ Nov 22 2012, 01:33) Ther...   Nov 22 2012, 18:10
- - Kamedo2   Bitrate vs Score plot of the 20 samples used. Opu...   Nov 21 2012, 03:26
- - Dynamic   I think the objectives in tests (experiments) matt...   Nov 22 2012, 19:03
|- - Kamedo2   QUOTE (Dynamic @ Nov 23 2012, 03:03) Ques...   Nov 22 2012, 22:42
- - Kamedo2   QUOTE rjamorim: There's some inverse proportio...   Nov 23 2012, 16:35
- - Kamedo2   I measured an average bitrate over wide range of n...   Nov 23 2012, 21:51
|- - jmvalin   QUOTE (Kamedo2 @ Nov 23 2012, 15:51) The ...   Nov 24 2012, 19:49
|- - Kamedo2   QUOTE (jmvalin @ Nov 25 2012, 03:49) QUOT...   Nov 25 2012, 15:34
- - Kamedo2   My post #34 might be too difficult. I wish I had b...   Nov 25 2012, 21:35
- - IgorC   Interesting. The Opus'es scores have less devi...   Nov 26 2012, 03:08
|- - Kamedo2   QUOTE (IgorC @ Nov 26 2012, 11:08) The Op...   Nov 26 2012, 23:56
|- - DonP   QUOTE (IgorC @ Nov 25 2012, 21:08) In thi...   Jan 3 2013, 02:57
- - Dynamic   Once again, Kamedo2, I applaud you for your testin...   Nov 26 2012, 21:57
|- - Kamedo2   QUOTE (Dynamic @ Nov 27 2012, 05:57) As I...   Nov 27 2012, 00:14
- - jmvalin   Kamedo2, can you give 1.1-alpha a try? It includes...   Jan 3 2013, 01:09
- - Kamedo2   QUOTE (jmvalin @ Jan 3 2013, 09:09) Kamed...   Jan 5 2013, 11:50


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 22nd December 2014 - 12:02