IPB

Welcome Guest ( Log In | Register )

Codecs and settings for 64kbit/s SE listening test, criticism required
Serge Smirnoff
post Oct 14 2012, 21:55
Post #1





Group: Members
Posts: 371
Joined: 14-December 01
Member No.: 641



As SoundExpert now has pretty stable flow of volunteer testers it is time to update codecs in all bitrate sections. After a short discussion at Russian bit-torrent tracker we decided to update 64-96-128-192-256-320 sections first. For the 64kbit/s section five codecs were chosen:

Fhg AAC (2012-06-24) - 59.2 kbit/s (fhgaacenc --vbr 2 se_ref.wav)
QAAC TVBR (v1.42) - 59.6 kbit/s (qaac --he -v56 se_ref.wav)
Nero AAC (v1.5.4.0) - 60.1 kbit/s (neroAacEnc.exe -q 0.25 -if se_ref.wav -of out.mp4)
Vorbis (Xiph 1.3.3) - 60.3 kbit/s (oggenc2.exe -q-0.3 se_ref.wav)
Opus (libopus 1.0.1) - 59.9 kbit/s (opusenc --bitrate 59 se_ref48.wav out.opus)
Conversion chain for Opus: 44.1/16 -->> 48/24(Audition CS6) -->> opusenc -->> foobar2000(48/24) -->> 44.1/16(Audition CS6)

Bitrates are calculated on the basis of nine SE test samples concatenated. Fortunately first two codecs have close values of resulting bitrates at their corresponding discrete quality settings. Other contenders were adjusted to have close values.

Did we miss something?


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post
 
Start new topic
Replies
yourlord
post Oct 15 2012, 01:08
Post #2





Group: Members
Posts: 226
Joined: 1-March 11
Member No.: 88621



Sounds like a useless test to me. There is nothing to be gained by exposing otherwise inaudible artifacts. These codecs make decisions based on the fact the artifacts this test seeks to expose would be inaudible.

As for including mp3, it's still the most popular codec, and it would be useful to see the bitrate at which mp3 gains parity with the test cases.

This post has been edited by yourlord: Oct 15 2012, 01:10
Go to the top of the page
+Quote Post
Serge Smirnoff
post Oct 15 2012, 01:12
Post #3





Group: Members
Posts: 371
Joined: 14-December 01
Member No.: 641



QUOTE (yourlord @ Oct 15 2012, 04:08) *
Sounds like a useless test to me. There is nothing to be gained by exposing otherwise inaudible artifacts. These codecs make decisions based on the fact the artifacts this test seeks to expose would be inaudible.

Below 128kbit/s artifact amplification is not applied. Outputs of codecs are used as is.


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post
yourlord
post Oct 15 2012, 02:23
Post #4





Group: Members
Posts: 226
Joined: 1-March 11
Member No.: 88621



QUOTE (Serge Smirnoff @ Oct 14 2012, 20:12) *
Below 128kbit/s artifact amplification is not applied. Outputs of codecs are used as is.


The fact remains I question doing it above 128kbps.. Amplifying inaudible artifacts to the point they become audible SERVES NO PURPOSE. You can't judge the quality of a lossy codec that way. The whole exercise will do nothing but provide misleading reference material which nimrods will use to base inaccurate claims that some codec is better or worse than another because of the INAUDIBLE artifacts this test needlessly exposes.

Each codec should be included in all test samples. If for no other reason than to illustrate how badly mp3 works at those low bit rates compared to the more modern ones.

If every codec other than mp3 becomes transparent at 128kbps then let your study confirm that so that it adds even more to the mountain of evidence that above those bitrates you can use any codec you like with no audible problems, and that mp3 sucks. laugh.gif

This post has been edited by yourlord: Oct 15 2012, 03:06
Go to the top of the page
+Quote Post
Serge Smirnoff
post Oct 15 2012, 09:19
Post #5





Group: Members
Posts: 371
Joined: 14-December 01
Member No.: 641



QUOTE (yourlord @ Oct 15 2012, 05:23) *
The fact remains I question doing it above 128kbps.. Amplifying inaudible artifacts to the point they become audible SERVES NO PURPOSE. You can't judge the quality of a lossy codec that way. The whole exercise will do nothing but provide misleading reference material which nimrods will use to base inaccurate claims that some codec is better or worse than another because of the INAUDIBLE artifacts this test needlessly exposes.

Please not here, there is more appropriate place for the discussion - http://www.hydrogenaudio.org/forums/index....=85182&st=0

QUOTE (yourlord @ Oct 15 2012, 05:23) *
Each codec should be included in all test samples. If for no other reason than to illustrate how badly mp3 works at those low bit rates compared to the more modern ones.

I'm not sure there is a need to prove shortcomings of mp3 at low bitrates over and over again.


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post
Kohlrabi
post Oct 15 2012, 12:32
Post #6





Group: Super Moderator
Posts: 1084
Joined: 12-March 05
From: Kiel, Germany
Member No.: 20561



QUOTE (Serge Smirnoff @ Oct 15 2012, 10:19) *
QUOTE (yourlord @ Oct 15 2012, 05:23) *
The fact remains I question doing it above 128kbps.. Amplifying inaudible artifacts to the point they become audible SERVES NO PURPOSE. You can't judge the quality of a lossy codec that way. The whole exercise will do nothing but provide misleading reference material which nimrods will use to base inaccurate claims that some codec is better or worse than another because of the INAUDIBLE artifacts this test needlessly exposes.

Please not here, there is more appropriate place for the discussion - http://www.hydrogenaudio.org/forums/index....=85182&st=0
While the concerns with the basic premise have not been addressed as far as I can tell, this is the thread you asked for input and comments in this thread, so let's stick with this one instead of reviving the old one.

QUOTE (Serge Smirnoff @ Oct 15 2012, 10:19) *
QUOTE (yourlord @ Oct 15 2012, 05:23) *
Each codec should be included in all test samples. If for no other reason than to illustrate how badly mp3 works at those low bit rates compared to the more modern ones.

I'm not sure there is a need to prove shortcomings of mp3 at low bitrates over and over again.
I'm on the other hand not sure if there is any need to prove "shortcomings" of lossy encoders by trying to inflate certain, previously inaudible, artifacts in a listening test. How do you make sure thise method doesn't artificially bias towards certain encoders/artifacts? If you want to prove that lossy encodes differ form the original, you're done now, since they obviously do and have to. Another useful metric to me is the binary issue of transparency. Either the (unaltered!) encoder result is transparent or it isn't. In the real world you'll never have weird mixes where you superimpose difference signals onto the encoded signal. This method is completely artificial with no real world application or meaning. Another thing done regularly here are the ABC tests, but those are mainly useful to grade encoders on results with obvious audible flaws, to decide which encoder produces the less annoying results.

This post has been edited by Kohlrabi: Oct 15 2012, 12:34


--------------------
Ceterum censeo Masterdiskem esse delendam.
Go to the top of the page
+Quote Post
saratoga
post Oct 15 2012, 16:10
Post #7





Group: Members
Posts: 5047
Joined: 2-September 02
Member No.: 3264



QUOTE (Kohlrabi @ Oct 15 2012, 07:32) *
How do you make sure thise method doesn't artificially bias towards certain encoders/artifacts? If you want to prove that lossy encodes differ form the original, you're done now, since they obviously do and have to. Another useful metric to me is the binary issue of transparency. Either the (unaltered!) encoder result is transparent or it isn't. In the real world you'll never have weird mixes where you superimpose difference signals onto the encoded signal. This method is completely artificial with no real world application or meaning.


To add to this point, I notice that the newer version of the site no longer ranks SBR codecs above non-SBR codecs, presumably due to some adjustment of the 'enhancement' process to give less obviously incorrect results?
Go to the top of the page
+Quote Post

Posts in this topic
- Serge Smirnoff   Codecs and settings for 64kbit/s SE listening test   Oct 14 2012, 21:55
- - lvqcl   BTW, qaac doesn't contain a codec: it uses AAC...   Oct 14 2012, 22:22
- - Kohlrabi   QUOTE (Serge Smirnoff @ Oct 14 2012, 22:5...   Oct 14 2012, 22:36
|- - Serge Smirnoff   QUOTE (Kohlrabi @ Oct 15 2012, 01:36) Wou...   Oct 15 2012, 00:49
- - eahm   QUOTE (Serge Smirnoff @ Oct 14 2012, 13:5...   Oct 14 2012, 22:42
|- - Serge Smirnoff   QUOTE (eahm @ Oct 15 2012, 01:42) QUOTE (...   Oct 15 2012, 00:40
- - lvqcl   QUOTE (Kohlrabi @ Oct 15 2012, 01:36) It...   Oct 14 2012, 22:47
|- - Kohlrabi   QUOTE (lvqcl @ Oct 14 2012, 23:47) QUOTE ...   Oct 14 2012, 22:55
|- - saratoga   QUOTE (Kohlrabi @ Oct 14 2012, 17:55) How...   Oct 14 2012, 23:08
- - yourlord   Sounds like a useless test to me. There is nothing...   Oct 15 2012, 01:08
|- - Serge Smirnoff   QUOTE (yourlord @ Oct 15 2012, 04:08) Sou...   Oct 15 2012, 01:12
||- - yourlord   QUOTE (Serge Smirnoff @ Oct 14 2012, 20:1...   Oct 15 2012, 02:23
||- - Serge Smirnoff   QUOTE (yourlord @ Oct 15 2012, 05:23) The...   Oct 15 2012, 09:19
||- - Kohlrabi   QUOTE (Serge Smirnoff @ Oct 15 2012, 10:1...   Oct 15 2012, 12:32
||- - Serge Smirnoff   QUOTE (Kohlrabi @ Oct 15 2012, 15:32) I...   Oct 15 2012, 13:34
|||- - yourlord   QUOTE (Serge Smirnoff @ Oct 15 2012, 08:3...   Oct 15 2012, 17:38
|||- - Serge Smirnoff   QUOTE (yourlord @ Oct 15 2012, 20:38) QUO...   Oct 16 2012, 01:17
||- - saratoga   QUOTE (Kohlrabi @ Oct 15 2012, 07:32) How...   Oct 15 2012, 16:10
||- - Serge Smirnoff   QUOTE (saratoga @ Oct 15 2012, 19:10) QUO...   Oct 15 2012, 17:30
|- - Serge Smirnoff   QUOTE (yourlord @ Oct 15 2012, 04:08) As ...   Oct 15 2012, 01:19
- - LithosZA   What I would like to see one day is more CVBR test...   Oct 15 2012, 14:43
|- - Serge Smirnoff   QUOTE (LithosZA @ Oct 15 2012, 17:43) Wha...   Oct 15 2012, 17:15
- - Kohlrabi   In short, you need to understand what lossy audio/...   Oct 15 2012, 18:22
- - LithosZA   QUOTE All AAC contenders for this 64kbit/s testing...   Oct 15 2012, 20:57
|- - Serge Smirnoff   QUOTE (LithosZA @ Oct 15 2012, 23:57) QUO...   Oct 16 2012, 02:27
- - yourlord   How about, can we have a test worth performing? I...   Oct 15 2012, 21:55
|- - Serge Smirnoff   QUOTE (yourlord @ Oct 16 2012, 00:55) How...   Oct 16 2012, 01:42
|- - yourlord   QUOTE (Serge Smirnoff @ Oct 15 2012, 20:4...   Oct 16 2012, 03:21
- - greynol   This argument is nothing new here (I too am skepti...   Oct 15 2012, 22:01
- - Serge Smirnoff   The following codecs were added to 64kbit/s sectio...   Oct 28 2012, 10:32
- - C.R.Helmrich   Thanks! How does the reliability rating work? ...   Oct 28 2012, 17:43
|- - Serge Smirnoff   QUOTE (C.R.Helmrich @ Oct 28 2012, 20:43)...   Oct 28 2012, 19:49
|- - skamp   QUOTE (Serge Smirnoff @ Oct 28 2012, 19:4...   Oct 28 2012, 20:03
|- - Serge Smirnoff   QUOTE (skamp @ Oct 28 2012, 23:03) QUOTE ...   Oct 28 2012, 20:11
- - LithosZA   That must be one good MP3 encoder Who knew MP3 64...   Oct 28 2012, 19:21
- - Serge Smirnoff   It turned out that reliability of ratings in 64kbi...   Oct 29 2012, 02:14
- - Xanikseo   One thing I can't comprehend, is why each of t...   Oct 31 2012, 00:21
|- - eahm   QUOTE (Xanikseo @ Oct 30 2012, 16:21) One...   Oct 31 2012, 02:20
- - jensend   QUOTE (Serge Smirnoff @ Oct 14 2012, 14:5...   Oct 31 2012, 05:49
- - Serge Smirnoff   Bitrate issue: they are calculated on the basis of...   Oct 31 2012, 09:31
- - Xanikseo   I am also generally suspicious of the other result...   Oct 31 2012, 15:25
- - Remedial Sound   It would seem there are 10 people in the world, th...   Oct 31 2012, 16:03
- - Serge Smirnoff   Detailed results of this listening test are availa...   Apr 11 2013, 23:01
- - greynol   QUOTE (greynol @ Oct 15 2012, 14:01) I to...   Apr 13 2013, 00:25
- - LithosZA   From this I can gather that Vorbis should give abo...   Apr 13 2013, 07:10
- - Serge Smirnoff   Thanks, greynol. I still have some questions conc...   Apr 13 2013, 13:16
- - Serge Smirnoff   Below are results using slightly different statist...   Nov 28 2013, 12:23
- - Serge Smirnoff   Following this very painful but insightful discuss...   Mar 29 2014, 21:53


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 24th October 2014 - 17:27