IPB

Welcome Guest ( Log In | Register )

Personal Listening Test of Opus, Celt, AAC at 75-100kbps, ABC/HR blind test, 1 Listener
Kamedo2
post Nov 17 2012, 09:25
Post #1





Group: Members
Posts: 219
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



Abstract:
Blind Comparison between 2012/09 new Opusenc(tfsel5), old Celtenc 0.11.2, Apple AAC-LC tvbr, cvbr.
This is an English version of my original post in Japanese. http://d.hatena.ne.jp/kamedo2/20121116/1353099244#seemore

Encoders:
libopus 0.9.11-146-gdc4f83b-exp_analysis
https://people.xiph.org/~greg/opus-tools_exp_dc4f83be.zip
celt-0.11.2-win32
https://people.xiph.org/~greg/celt-0.11.2-win32.zip
qaac 1.40
qaac 1.40

Settings:
opusenc --bitrate 66 input.wav output.wav
celtenc input.48k.raw --bitrate 75 --comp 10 output.wav
qaac --cvbr 72 -o output.m4a input.wav
qaac --tvbr 27 -o output.m4a input.wav
opusenc --bitrate 90 input.wav output.wav
celtenc input.48k.raw --bitrate 100 --comp 10 output.wav
qaac --cvbr 96 -o output.m4a input.wav
qaac --tvbr 45 -o output.m4a input.wav

Samples:
20 Sounds of various genres, from easy to modestly critical.
http://zak.s206.xrea.com/bitratetest/main.htm
To download, access to the link above, 2nd paragraph, 3rd-6th links. (40_30sec - Run up)

Hardwares:
Sony PSP-3000 + RP-HT560(1st) , RP-HJE150(2nd), took the average of the two results.

Results:




Conclusions & Observations:
I could not detect a significant improvement in the new September 1st version of Opus, from the old Celtenc in 2011.
It's possibly because the new Opus inflates bitrates more than it improves qualities, although the set of sounds contain easy samples.
On 75kbps, Opus/Celt are markedly better. On 100kbps, there is no big difference between those codecs.

Raw data:
40 Logs and encoders, decoders log
http://zak.s206.xrea.com/bitratetest/log_o...kbps100kbps.zip
CODE
% Opus, AAC 75kbps, 100kbps ABC/HR Score
% This format is compatible with my graphmaker, as well as ff123's FRIEDMAN.
opus_75k    celt_75k    cvbr_75k    tvbr_75k    opus100k    celt100k    cvbr100k    tvbr100k
%features 6 75kbps 75kbps 75kbps 75kbps 100kbps 100kbps 100kbps 100kbps
%features 7 OPUS OPUS AAC-LC AAC-LC OPUS OPUS AAC-LC AAC-LC
3.050    3.100    2.500    2.750    3.500    3.750    3.700    3.800    
3.750    2.950    2.700    2.750    4.050    3.800    4.000    3.950    
2.800    2.550    3.000    3.000    3.600    3.250    4.050    3.900    
2.700    3.150    2.350    2.300    3.350    3.800    3.600    3.700    
4.000    3.400    2.850    2.850    4.350    3.900    3.550    3.550    
2.600    2.550    2.800    2.800    3.350    3.150    3.950    3.900    
3.400    3.950    3.000    3.200    3.850    4.500    3.700    3.800    
3.450    3.500    2.900    2.800    3.850    4.050    4.050    4.150    
2.950    2.700    3.550    3.450    3.250    3.450    4.000    3.850    
3.100    3.400    2.750    2.600    3.800    3.850    4.150    4.000    
3.350    3.100    2.600    2.600    3.750    3.400    3.450    3.500    
3.750    3.350    2.800    2.950    4.050    3.750    3.800    3.850    
3.550    3.300    2.600    2.650    4.250    3.950    3.750    3.600    
3.100    3.350    2.750    2.550    3.650    3.700    3.850    3.800    
3.400    3.450    2.900    2.900    3.650    3.950    3.750    3.900    
3.250    3.300    2.750    2.800    3.650    3.850    3.950    3.750    
3.600    3.800    3.300    3.300    3.550    4.000    3.650    3.700    
3.700    3.350    3.300    3.300    3.900    3.650    4.100    4.000    
3.100    3.600    3.150    3.000    3.700    3.800    4.100    3.850    
3.650    4.050    3.000    2.900    4.050    4.250    3.750    3.550

It's not strange that some scores get 0.050 scale because I did tests twice per each music.
Go to the top of the page
+Quote Post
 
Start new topic
Replies
Dynamic
post Nov 18 2012, 09:20
Post #2





Group: Members
Posts: 813
Joined: 17-September 06
Member No.: 35307



QUOTE
x-axis=actual bitrate


That was one query I did have about your method, Kamedo2 - mainly because I don't speak more than about 5 words of Japanese to find out for myself.

As you have been so thorough, I expect we can assume that you tested the encoders on a larger collection (maybe 10 or many more CDs) of normal music to determine the target settings to achieve the bitrate target, and that you plotted THAT collection-wide bitrate (for the large sample size) on the horizontal axis of your graphs, ignoring the actual bitrate used for the test samples?

The bitrate on a number of short samples (with more problem samples than usual) may properly be much higher than the average of a wide collection and is not relevant in the context of fair comparison. (That's the beauty of well tuned VBR, and to some extent modern constrained VBR as used by the 'CBR' modes - using much higher bitrate when the sound requires it and less when it doesn't without inflating the average over a whole collection).

There's also a chance that Opus produces lower bitrates for short samples as it doesn't use large codebooks compared to AAC, Vorbis etc., so using the sample-wide bitrate would give an over-emphasised disadvantage to AAC at lower bitrates on shorter-than-normal samples, again requiring the bitrate of a wider collection of representative CD releases, presumably encoded in the popular file-per-track mode without any pre-applied Replay Gain, as that's most representative of real use of lossy encoders for playback.
Go to the top of the page
+Quote Post
Kamedo2
post Nov 18 2012, 10:23
Post #3





Group: Members
Posts: 219
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



QUOTE (Dynamic @ Nov 18 2012, 17:20) *
QUOTE
x-axis=actual bitrate


That was one query I did have about your method, Kamedo2 - mainly because I don't speak more than about 5 words of Japanese to find out for myself.

As you have been so thorough, I expect we can assume that you tested the encoders on a larger collection (maybe 10 or many more CDs) of normal music to determine the target settings to achieve the bitrate target, and that you plotted THAT collection-wide bitrate (for the large sample size) on the horizontal axis of your graphs, ignoring the actual bitrate used for the test samples?


Horizontal axis is always the average actual bitrate including headers and footers of each sample.
I calculated filesize*8/(sample_num/44100) of each sample, typically 20-30sec.
Then, I took the average of the 20 bitrates in bps. Larger collections are NEVER used.
Encoded files are typically around 200KB. Opus should benefit form smaller codebooks in this case.

QUOTE
larger collection (maybe 10 or many more CDs) of normal music

My set of samples are fairly normal. At least they are not an array of super-critical fatboys, although short.
The last 5 samples from the 20 samples I used.(Reunion Blues ~ Run up)
http://zak.s206.xrea.com/bitratetest/bitra...st_wav25-29.zip

CODE
list of audio file size in KB(1000B)
286 377 187 220 231 151 153 278 104 232 252 204 182 246 187 251 267 255 155 170
283 278 182 231 175 165 191 264 100 237 260 192 188 265 190 282 289 256 186 189
303 280 202 233 190 163 191 266 108 233 273 199 188 263 184 313 278 256 187 189
319 216 208 224 175 163 191 253 108 209 268 205 163 240 154 341 233 242 169 158
389 509 252 298 313 204 202 375 139 311 342 276 244 334 238 341 344 344 210 229
379 373 242 309 234 220 255 353 133 316 346 257 250 354 253 376 386 343 249 252
415 370 265 310 252 222 252 355 143 312 372 264 248 354 249 411 366 341 250 245
425 287 275 310 239 222 262 347 143 282 372 282 231 340 212 455 305 330 241 222

list of audio file bitrate including headers and footers in kbps(1000bps)
  76 101  78  72  92  69  60  79  80  74  69  81  73  69  75  67  72  74  63  68
  75  74  76  76  70  76  75  75  77  76  72  76  75  75  76  75  78  75  75  76
  81  75  84  76  76  75  75  75  84  74  75  79  75  74  74  84  75  75  75  75
  85  58  87  73  70  75  75  72  84  67  74  81  65  68  62  91  63  71  68  63
104 136 105  97 125  93  79 106 107  99  94 110  98  94  96  91  93 100  84  92
101 100 101 101  94 101 100 100 103 101  95 102 100 100 102 100 104 100 100 101
111  99 111 101 101 102  99 100 111  99 102 105  99 100 100 110  99  99 101  98
113  77 115 101  96 102 103  98 111  90 102 112  92  96  86 121  83  96  97  89
Go to the top of the page
+Quote Post

Posts in this topic
- Kamedo2   Personal Listening Test of Opus, Celt, AAC at 75-100kbps   Nov 17 2012, 09:25
- - C.R.Helmrich   Thanks for this interesting test, Kamedo, and welc...   Nov 17 2012, 10:28
|- - Kamedo2   QUOTE (C.R.Helmrich @ Nov 17 2012, 18:28)...   Nov 17 2012, 10:40
- - Dynamic   Thank you for your time and dedication, Kamedo2 ...   Nov 17 2012, 10:48
- - IgorC   Kamedo2, Thank You for all your tests. Glad to see...   Nov 17 2012, 11:04
- - Kamedo2   The samples I used The ABX criteria is 12/15(p=0...   Nov 17 2012, 18:44
- - Anakunda   QUOTE (Kamedo2 @ Nov 17 2012, 09:25) Blin...   Nov 17 2012, 23:45
|- - C.R.Helmrich   QUOTE (Anakunda @ Nov 18 2012, 00:45) Is ...   Nov 18 2012, 00:54
|- - Kamedo2   QUOTE (Anakunda @ Nov 18 2012, 07:45) Is ...   Nov 18 2012, 05:47
- - Dynamic   QUOTE x-axis=actual bitrate That was one query I ...   Nov 18 2012, 09:20
|- - Kamedo2   QUOTE (Dynamic @ Nov 18 2012, 17:20) QUOT...   Nov 18 2012, 10:23
- - Dynamic   Thank you for the clarification. It seems that du...   Nov 18 2012, 20:07
- - IgorC   Kamedo2, I'm not here to criticize your test...   Nov 18 2012, 20:12
|- - Kamedo2   QUOTE (IgorC @ Nov 19 2012, 04:12) As all...   Nov 19 2012, 02:17
|- - jmvalin   Hi Kamedo2, thanks for the test. From what I see, ...   Nov 19 2012, 20:03
- - lvqcl   I took my Opus compile (libopus v1.0.1-140-gc55f4d...   Nov 19 2012, 20:32
|- - Kamedo2   QUOTE (jmvalin @ Nov 20 2012, 04:03) it w...   Nov 19 2012, 23:19
|- - jmvalin   QUOTE (Kamedo2 @ Nov 19 2012, 17:19) Acco...   Nov 20 2012, 18:39
- - Dynamic   Thanks again to everyone in this thread. I'm c...   Nov 20 2012, 03:03
|- - jmvalin   QUOTE (Dynamic @ Nov 19 2012, 21:03) We d...   Nov 20 2012, 05:13
- - C.R.Helmrich   QUOTE (Kamedo2 @ Nov 17 2012, 10:25) Samp...   Nov 20 2012, 23:01
- - Kamedo2   QUOTE (jmvalin @ Nov 20 2012, 13:13) CELT...   Nov 21 2012, 02:08
|- - jmvalin   QUOTE (Kamedo2 @ Nov 20 2012, 20:08) I as...   Nov 21 2012, 03:15
|- - Kamedo2   QUOTE (jmvalin @ Nov 21 2012, 11:15) Actu...   Nov 21 2012, 05:37
|- - jmvalin   QUOTE (Kamedo2 @ Nov 20 2012, 23:37) I me...   Nov 22 2012, 02:45
|- - Kamedo2   QUOTE (jmvalin @ Nov 22 2012, 10:45) QUOT...   Nov 22 2012, 07:33
|- - jmvalin   QUOTE (Kamedo2 @ Nov 22 2012, 01:33) Ther...   Nov 22 2012, 18:10
- - Kamedo2   Bitrate vs Score plot of the 20 samples used. Opu...   Nov 21 2012, 03:26
- - Dynamic   I think the objectives in tests (experiments) matt...   Nov 22 2012, 19:03
|- - Kamedo2   QUOTE (Dynamic @ Nov 23 2012, 03:03) Ques...   Nov 22 2012, 22:42
- - Kamedo2   QUOTE rjamorim: There's some inverse proportio...   Nov 23 2012, 16:35
- - Kamedo2   I measured an average bitrate over wide range of n...   Nov 23 2012, 21:51
|- - jmvalin   QUOTE (Kamedo2 @ Nov 23 2012, 15:51) The ...   Nov 24 2012, 19:49
|- - Kamedo2   QUOTE (jmvalin @ Nov 25 2012, 03:49) QUOT...   Nov 25 2012, 15:34
- - Kamedo2   My post #34 might be too difficult. I wish I had b...   Nov 25 2012, 21:35
- - IgorC   Interesting. The Opus'es scores have less devi...   Nov 26 2012, 03:08
|- - Kamedo2   QUOTE (IgorC @ Nov 26 2012, 11:08) The Op...   Nov 26 2012, 23:56
|- - DonP   QUOTE (IgorC @ Nov 25 2012, 21:08) In thi...   Jan 3 2013, 02:57
- - Dynamic   Once again, Kamedo2, I applaud you for your testin...   Nov 26 2012, 21:57
|- - Kamedo2   QUOTE (Dynamic @ Nov 27 2012, 05:57) As I...   Nov 27 2012, 00:14
- - jmvalin   Kamedo2, can you give 1.1-alpha a try? It includes...   Jan 3 2013, 01:09
- - Kamedo2   QUOTE (jmvalin @ Jan 3 2013, 09:09) Kamed...   Jan 5 2013, 11:50


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 16th September 2014 - 13:00