IPB

Welcome Guest ( Log In | Register )

2 Pages V   1 2 >  
Reply to this topicStart new topic
Codecs and settings for 64kbit/s SE listening test, criticism required
Serge Smirnoff
post Oct 14 2012, 21:55
Post #1





Group: Members
Posts: 370
Joined: 14-December 01
Member No.: 641



As SoundExpert now has pretty stable flow of volunteer testers it is time to update codecs in all bitrate sections. After a short discussion at Russian bit-torrent tracker we decided to update 64-96-128-192-256-320 sections first. For the 64kbit/s section five codecs were chosen:

Fhg AAC (2012-06-24) - 59.2 kbit/s (fhgaacenc --vbr 2 se_ref.wav)
QAAC TVBR (v1.42) - 59.6 kbit/s (qaac --he -v56 se_ref.wav)
Nero AAC (v1.5.4.0) - 60.1 kbit/s (neroAacEnc.exe -q 0.25 -if se_ref.wav -of out.mp4)
Vorbis (Xiph 1.3.3) - 60.3 kbit/s (oggenc2.exe -q-0.3 se_ref.wav)
Opus (libopus 1.0.1) - 59.9 kbit/s (opusenc --bitrate 59 se_ref48.wav out.opus)
Conversion chain for Opus: 44.1/16 -->> 48/24(Audition CS6) -->> opusenc -->> foobar2000(48/24) -->> 44.1/16(Audition CS6)

Bitrates are calculated on the basis of nine SE test samples concatenated. Fortunately first two codecs have close values of resulting bitrates at their corresponding discrete quality settings. Other contenders were adjusted to have close values.

Did we miss something?


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post
lvqcl
post Oct 14 2012, 22:22
Post #2





Group: Developer
Posts: 3381
Joined: 2-December 07
Member No.: 49183



BTW, qaac doesn't contain a codec: it uses AAC encoder from iTunes. So probably it's better to write:
QAAC 1.42 + iTunes 10.7
or something like this.
Go to the top of the page
+Quote Post
Kohlrabi
post Oct 14 2012, 22:36
Post #3





Group: Super Moderator
Posts: 1046
Joined: 12-March 05
From: Kiel, Germany
Member No.: 20561



QUOTE (Serge Smirnoff @ Oct 14 2012, 22:55) *
As SoundExpert now has pretty stable flow of volunteer testers it is time to update codecs in all bitrate sections. After a short discussion at Russian bit-torrent tracker we decided to update 64-96-128-192-256-320 sections first.
It's probably a very daunting task to get good results going beyond 128 kbps. Have you performed successful listening tests at high bitrates in the past?

Wouldn't it be interesting to include MP3/LAME instead of three flavours of AAC?

This post has been edited by Kohlrabi: Oct 14 2012, 22:52


--------------------
Ceterum censeo Masterdiskem esse delendam.
Go to the top of the page
+Quote Post
eahm
post Oct 14 2012, 22:42
Post #4





Group: Members
Posts: 1079
Joined: 11-February 12
Member No.: 97076



QUOTE (Serge Smirnoff @ Oct 14 2012, 13:55) *
QAAC TVBR (v1.42) - 59.6 kbit/s (qaac --he -v56 se_ref.wav)

-v is not TVBR but CVBR. -V is TVBR and --he is not possible with it.

qaac TVBR settings:

CODE
Q0 - Q4 (0) = ~40 Kbps
Q5 - Q13 (9) = ~45 Kbps
Q14 - Q22 (18) = ~75 Kbps
Q23 - Q31 (27) = ~80 Kbps
Q32 - Q40 (36) = ~95 Kbps
Q41 - Q49 (45) = ~105 Kbps
Q50 - Q58 (54) = ~115 Kbps
Q59 - Q68 (63) = ~135 Kbps
Q69 - Q77 (73) = ~150 Kbps
Q78 - Q86 (82) = ~165 Kbps
Q87 - Q95 (91) = ~195 Kbps
Q96 - Q104 (100) = ~225 Kbps
Q105 - Q113 (109) = ~255 Kbps
Q114 - Q122 (118) = ~285 Kbps
Q123 - Q127 (127) = ~320 Kbps


QUOTE (lvqcl @ Oct 14 2012, 14:22) *
BTW, qaac doesn't contain a codec: it uses AAC encoder from iTunes. So probably it's better to write:
QAAC 1.42 + iTunes 10.7
or something like this.

Even better if he adds the CoreAudioToolbox.dll version he's using it (7.9.7.9, 7.9.8.1 etc.).

This post has been edited by eahm: Oct 14 2012, 22:45


--------------------
/lwAsIimz
Go to the top of the page
+Quote Post
lvqcl
post Oct 14 2012, 22:47
Post #5





Group: Developer
Posts: 3381
Joined: 2-December 07
Member No.: 49183



QUOTE (Kohlrabi @ Oct 15 2012, 01:36) *
It's probably a very daunting task to get good results going beyond 128 kbps. Have you performed successful listening tests at high bitrates in the past?

QUOTE
Ratings above 5-th grade mean that the devices/technologies have some quality headroom, their artifacts are beyond threshold of human audibility. Testing files of such devices are processed additionally - sound artifacts are amplified to the extent when they could be heard by ordinary listeners.

(from soundexpert.org, emphasis mine)

This post has been edited by lvqcl: Oct 14 2012, 22:48
Go to the top of the page
+Quote Post
Kohlrabi
post Oct 14 2012, 22:55
Post #6





Group: Super Moderator
Posts: 1046
Joined: 12-March 05
From: Kiel, Germany
Member No.: 20561



QUOTE (lvqcl @ Oct 14 2012, 23:47) *
QUOTE (Kohlrabi @ Oct 15 2012, 01:36) *
It's probably a very daunting task to get good results going beyond 128 kbps. Have you performed successful listening tests at high bitrates in the past?

QUOTE
Ratings above 5-th grade mean that the devices/technologies have some quality headroom, their artifacts are beyond threshold of human audibility. Testing files of such devices are processed additionally - sound artifacts are amplified to the extent when they could be heard by ordinary listeners.

(from soundexpert.org, emphasis mine)
How do you algorithmically selectively amplify artifacts, other than modifying the encoders to work worse than they'd normally would?


--------------------
Ceterum censeo Masterdiskem esse delendam.
Go to the top of the page
+Quote Post
saratoga
post Oct 14 2012, 23:08
Post #7





Group: Members
Posts: 4957
Joined: 2-September 02
Member No.: 3264



QUOTE (Kohlrabi @ Oct 14 2012, 17:55) *
How do you algorithmically selectively amplify artifacts, other than modifying the encoders to work worse than they'd normally would?


IIRC they subtract the encoded signal from the lossless input and then do some kind of processing on it and then add it back to "enhance" artifacts. The end result is that the encoded files are easier to distinguish from lossless, but OTOH I don't think anyone has ever shown that the differences correlate with actual audio quality.
Go to the top of the page
+Quote Post
Serge Smirnoff
post Oct 15 2012, 00:40
Post #8





Group: Members
Posts: 370
Joined: 14-December 01
Member No.: 641



QUOTE (eahm @ Oct 15 2012, 01:42) *
QUOTE (Serge Smirnoff @ Oct 14 2012, 13:55) *
QAAC TVBR (v1.42) - 59.6 kbit/s (qaac --he -v56 se_ref.wav)

-v is not TVBR but CVBR. -V is TVBR and --he is not possible with it.

accepted, thanks; for 64kbit/s I think HE-AAC is more appropriate setting for this codec. True VBR will be used at higher bitrates.

QUOTE (eahm @ Oct 15 2012, 01:42) *
QUOTE (lvqcl @ Oct 14 2012, 14:22) *
BTW, qaac doesn't contain a codec: it uses AAC encoder from iTunes. So probably it's better to write:
QAAC 1.42 + iTunes 10.7
or something like this.

Even better if he adds the CoreAudioToolbox.dll version he's using it (7.9.7.9, 7.9.8.1 etc.).

QAAC 1.42 + iTunes 10.7.0.21 (CoreAudioToolbox.dll 7.9.7.3)
This CoreAudioToolbox looks outdated but iTunes and QuickTime are the latest on my system. Is it important to have the latest version of the file?


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post
Serge Smirnoff
post Oct 15 2012, 00:49
Post #9





Group: Members
Posts: 370
Joined: 14-December 01
Member No.: 641



QUOTE (Kohlrabi @ Oct 15 2012, 01:36) *
Wouldn't it be interesting to include MP3/LAME instead of three flavours of AAC?

I think 64 is too low for mp3, result is too predictable and the usage scenario is too uncommon.


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post
yourlord
post Oct 15 2012, 01:08
Post #10





Group: Members
Posts: 217
Joined: 1-March 11
Member No.: 88621



Sounds like a useless test to me. There is nothing to be gained by exposing otherwise inaudible artifacts. These codecs make decisions based on the fact the artifacts this test seeks to expose would be inaudible.

As for including mp3, it's still the most popular codec, and it would be useful to see the bitrate at which mp3 gains parity with the test cases.

This post has been edited by yourlord: Oct 15 2012, 01:10
Go to the top of the page
+Quote Post
Serge Smirnoff
post Oct 15 2012, 01:12
Post #11





Group: Members
Posts: 370
Joined: 14-December 01
Member No.: 641



QUOTE (yourlord @ Oct 15 2012, 04:08) *
Sounds like a useless test to me. There is nothing to be gained by exposing otherwise inaudible artifacts. These codecs make decisions based on the fact the artifacts this test seeks to expose would be inaudible.

Below 128kbit/s artifact amplification is not applied. Outputs of codecs are used as is.


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post
Serge Smirnoff
post Oct 15 2012, 01:19
Post #12





Group: Members
Posts: 370
Joined: 14-December 01
Member No.: 641



QUOTE (yourlord @ Oct 15 2012, 04:08) *
As for including mp3, it's still the most popular codec, and it would be useful to see the bitrate at which mp3 gains parity with the test cases.

Preliminary decision was to start testing of mp3 from 96kbit/s, should we really begin with 64kbit/s? There are a lot of outdated mp3 codecs in this section already, btw.


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post
yourlord
post Oct 15 2012, 02:23
Post #13





Group: Members
Posts: 217
Joined: 1-March 11
Member No.: 88621



QUOTE (Serge Smirnoff @ Oct 14 2012, 20:12) *
Below 128kbit/s artifact amplification is not applied. Outputs of codecs are used as is.


The fact remains I question doing it above 128kbps.. Amplifying inaudible artifacts to the point they become audible SERVES NO PURPOSE. You can't judge the quality of a lossy codec that way. The whole exercise will do nothing but provide misleading reference material which nimrods will use to base inaccurate claims that some codec is better or worse than another because of the INAUDIBLE artifacts this test needlessly exposes.

Each codec should be included in all test samples. If for no other reason than to illustrate how badly mp3 works at those low bit rates compared to the more modern ones.

If every codec other than mp3 becomes transparent at 128kbps then let your study confirm that so that it adds even more to the mountain of evidence that above those bitrates you can use any codec you like with no audible problems, and that mp3 sucks. laugh.gif

This post has been edited by yourlord: Oct 15 2012, 03:06
Go to the top of the page
+Quote Post
Serge Smirnoff
post Oct 15 2012, 09:19
Post #14





Group: Members
Posts: 370
Joined: 14-December 01
Member No.: 641



QUOTE (yourlord @ Oct 15 2012, 05:23) *
The fact remains I question doing it above 128kbps.. Amplifying inaudible artifacts to the point they become audible SERVES NO PURPOSE. You can't judge the quality of a lossy codec that way. The whole exercise will do nothing but provide misleading reference material which nimrods will use to base inaccurate claims that some codec is better or worse than another because of the INAUDIBLE artifacts this test needlessly exposes.

Please not here, there is more appropriate place for the discussion - http://www.hydrogenaudio.org/forums/index....=85182&st=0

QUOTE (yourlord @ Oct 15 2012, 05:23) *
Each codec should be included in all test samples. If for no other reason than to illustrate how badly mp3 works at those low bit rates compared to the more modern ones.

I'm not sure there is a need to prove shortcomings of mp3 at low bitrates over and over again.


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post
Kohlrabi
post Oct 15 2012, 12:32
Post #15





Group: Super Moderator
Posts: 1046
Joined: 12-March 05
From: Kiel, Germany
Member No.: 20561



QUOTE (Serge Smirnoff @ Oct 15 2012, 10:19) *
QUOTE (yourlord @ Oct 15 2012, 05:23) *
The fact remains I question doing it above 128kbps.. Amplifying inaudible artifacts to the point they become audible SERVES NO PURPOSE. You can't judge the quality of a lossy codec that way. The whole exercise will do nothing but provide misleading reference material which nimrods will use to base inaccurate claims that some codec is better or worse than another because of the INAUDIBLE artifacts this test needlessly exposes.

Please not here, there is more appropriate place for the discussion - http://www.hydrogenaudio.org/forums/index....=85182&st=0
While the concerns with the basic premise have not been addressed as far as I can tell, this is the thread you asked for input and comments in this thread, so let's stick with this one instead of reviving the old one.

QUOTE (Serge Smirnoff @ Oct 15 2012, 10:19) *
QUOTE (yourlord @ Oct 15 2012, 05:23) *
Each codec should be included in all test samples. If for no other reason than to illustrate how badly mp3 works at those low bit rates compared to the more modern ones.

I'm not sure there is a need to prove shortcomings of mp3 at low bitrates over and over again.
I'm on the other hand not sure if there is any need to prove "shortcomings" of lossy encoders by trying to inflate certain, previously inaudible, artifacts in a listening test. How do you make sure thise method doesn't artificially bias towards certain encoders/artifacts? If you want to prove that lossy encodes differ form the original, you're done now, since they obviously do and have to. Another useful metric to me is the binary issue of transparency. Either the (unaltered!) encoder result is transparent or it isn't. In the real world you'll never have weird mixes where you superimpose difference signals onto the encoded signal. This method is completely artificial with no real world application or meaning. Another thing done regularly here are the ABC tests, but those are mainly useful to grade encoders on results with obvious audible flaws, to decide which encoder produces the less annoying results.

This post has been edited by Kohlrabi: Oct 15 2012, 12:34


--------------------
Ceterum censeo Masterdiskem esse delendam.
Go to the top of the page
+Quote Post
Serge Smirnoff
post Oct 15 2012, 13:34
Post #16





Group: Members
Posts: 370
Joined: 14-December 01
Member No.: 641



QUOTE (Kohlrabi @ Oct 15 2012, 15:32) *
I'm on the other hand not sure if there is any need to prove "shortcomings" of lossy encoders by trying to inflate certain, previously inaudible, artifacts in a listening test.

At 64kbit/s there is no need for artifacts amplification for sure. Above 128kbit/s meaningful results of ABX testing become more and more expensive (but still meaningful). SoundExpert proposes methodology that makes those tests less expensive. SE quality ratings of devices with small impairments could be considered as results of specially simplified listening tests. Results are experimental which is clearly stated on the site.


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post
LithosZA
post Oct 15 2012, 14:43
Post #17





Group: Members
Posts: 196
Joined: 26-February 11
Member No.: 88525



What I would like to see one day is more CVBR tests. All the previous listening test I have seen were more concerned about offline storage than streaming.
Go to the top of the page
+Quote Post
saratoga
post Oct 15 2012, 16:10
Post #18





Group: Members
Posts: 4957
Joined: 2-September 02
Member No.: 3264



QUOTE (Kohlrabi @ Oct 15 2012, 07:32) *
How do you make sure thise method doesn't artificially bias towards certain encoders/artifacts? If you want to prove that lossy encodes differ form the original, you're done now, since they obviously do and have to. Another useful metric to me is the binary issue of transparency. Either the (unaltered!) encoder result is transparent or it isn't. In the real world you'll never have weird mixes where you superimpose difference signals onto the encoded signal. This method is completely artificial with no real world application or meaning.


To add to this point, I notice that the newer version of the site no longer ranks SBR codecs above non-SBR codecs, presumably due to some adjustment of the 'enhancement' process to give less obviously incorrect results?
Go to the top of the page
+Quote Post
Serge Smirnoff
post Oct 15 2012, 17:15
Post #19





Group: Members
Posts: 370
Joined: 14-December 01
Member No.: 641



QUOTE (LithosZA @ Oct 15 2012, 17:43) *
What I would like to see one day is more CVBR tests. All the previous listening test I have seen were more concerned about offline storage than streaming.

All AAC contenders for this 64kbit/s testing are in CVBR mode.


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post
Serge Smirnoff
post Oct 15 2012, 17:30
Post #20





Group: Members
Posts: 370
Joined: 14-December 01
Member No.: 641



QUOTE (saratoga @ Oct 15 2012, 19:10) *
QUOTE (Kohlrabi @ Oct 15 2012, 07:32) *
How do you make sure thise method doesn't artificially bias towards certain encoders/artifacts? If you want to prove that lossy encodes differ form the original, you're done now, since they obviously do and have to. Another useful metric to me is the binary issue of transparency. Either the (unaltered!) encoder result is transparent or it isn't. In the real world you'll never have weird mixes where you superimpose difference signals onto the encoded signal. This method is completely artificial with no real world application or meaning.


To add to this point, I notice that the newer version of the site no longer ranks SBR codecs above non-SBR codecs, presumably due to some adjustment of the 'enhancement' process to give less obviously incorrect results?

The only adjustment that was brought into operation last year is post-screening of incoming grades. The reason of instability of high bit-rate ratings (320+) is insufficient number of testing points and the problem still needs some research. Another SBR codec in 192 section never showed higher results.

This post has been edited by Serge Smirnoff: Oct 15 2012, 17:31


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post
yourlord
post Oct 15 2012, 17:38
Post #21





Group: Members
Posts: 217
Joined: 1-March 11
Member No.: 88621



QUOTE (Serge Smirnoff @ Oct 15 2012, 08:34) *
At 64kbit/s there is no need for artifacts amplification for sure. Above 128kbit/s meaningful results of ABX testing become more and more expensive (but still meaningful). SoundExpert proposes methodology that makes those tests less expensive. SE quality ratings of devices with small impairments could be considered as results of specially simplified listening tests. Results are experimental which is clearly stated on the site.


If meaningful results of ABX tests above 128kbps become more and more expensive it's because the codecs are doing their jobs and producing audibly transparent output. At a point where normal ABX results become statistically insignificant then transparency has been reached and we're done. Artificially altering encoder output to highlight normally inaudible artifacts of the encoding process and then trying to assign a quality to a codec based on those artificially accentuated normally inaudible artifacts is a USELSS process. It has no application in the real world, it means nothing, and the results obtained from such "tests" are useless noise and best ignored.

You might as well subtract the lossy output from the original, post the spectrograms, and start running around screaming the sky is falling and vinyl is better than digital..

This post has been edited by yourlord: Oct 15 2012, 17:41
Go to the top of the page
+Quote Post
Kohlrabi
post Oct 15 2012, 18:22
Post #22





Group: Super Moderator
Posts: 1046
Joined: 12-March 05
From: Kiel, Germany
Member No.: 20561



In short, you need to understand what lossy audio/video encoding tries to achieve. It aims to produce audibly/visibly artifact-free files, and not generally artifact-free files. That is what lossless compression is for.

This post has been edited by Kohlrabi: Oct 15 2012, 18:29


--------------------
Ceterum censeo Masterdiskem esse delendam.
Go to the top of the page
+Quote Post
LithosZA
post Oct 15 2012, 20:57
Post #23





Group: Members
Posts: 196
Joined: 26-February 11
Member No.: 88525



QUOTE
All AAC contenders for this 64kbit/s testing are in CVBR mode.

Can we also use Vorbis and Opus at CVBR rates?

Lets say we have Codec X and Y
X we do at VBR. With some difficult songs X jumps to 82Kbps for some sections. It is okay for offline, because the average might be around 67Kbit/s for all the songs encoded.
Y we do at CVBR. It stays at +/- 64Kbps.
If Codec X won the listening test people might think X is also better than Y when it comes to streaming (Like internet radio stations). This might not be the case, because the X simply used more bits with difficult sections.
Go to the top of the page
+Quote Post
yourlord
post Oct 15 2012, 21:55
Post #24





Group: Members
Posts: 217
Joined: 1-March 11
Member No.: 88621



How about, can we have a test worth performing?

If AAC, Vorbis, Opus, and MP3 are all statistically transparent at a given nominal bit rate then they are all audibly the SAME QUALITY at that bit rate. No one codec offers any audible benefit over the others at that point. There is nothing to gain in claiming to judge codec quality by adding distortion to their output and pretending it somehow matters in the real world.

I'm honestly not sure why this thread hasn't been locked/removed. It smells of snake oil and pixie dust, or at the least is ill-conceived. These "tests" of adulterated codec outputs offer us no relevant results on which to base any kind of rational discussion or decisions, other than how NOT to conduct a codec quality test. It can only serve to spread disinformation and ignorance. IMO it has no place on HA.

This post has been edited by yourlord: Oct 15 2012, 21:56
Go to the top of the page
+Quote Post
greynol
post Oct 15 2012, 22:01
Post #25





Group: Super Moderator
Posts: 10000
Joined: 1-April 04
From: San Francisco
Member No.: 13167



This argument is nothing new here (I too am skeptical about the relevance of SE tests). So long as TOS #8 or any other rule isn't being violated, the discussion can stand.

I was tempted to ask people to refrain from this line of conversation, but Serge did solicit criticism. I also think those who aren't familiar with SE should be aware that results from SE are not exactly in keeping with the spirit of this forum.


--------------------
I should publish a list of forum idiots.
Go to the top of the page
+Quote Post

2 Pages V   1 2 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 16th September 2014 - 02:38