IPB

Welcome Guest ( Log In | Register )

Public AAC Listening Test @ ~96 kbps [July 2011]: Results, Results and post-test discussion
IgorC
post Aug 23 2011, 19:56
Post #1





Group: Members
Posts: 1580
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



After the long time of preparations, discussions and realization of the test the results are finally here.

http://listening-tests.hydrogenaudio.org/i...-a/results.html

Summary: Apple won, FhG is the second, Coding Technologies is the third and Nero is the last

I appreciate all people who has supported the test and participated in it.

This post has been edited by IgorC: Aug 23 2011, 20:12
Go to the top of the page
+Quote Post
 
Start new topic
Replies
Sebastian Mares
post Aug 24 2011, 07:29
Post #2





Group: Members
Posts: 3637
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



It appears to me that the low anchor was way too bad. Shouldn't the low anchor be at around the same quality as the contenders, but "slightly" worse than all of them?


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
Garf
post Aug 24 2011, 09:35
Post #3


Server Admin


Group: Admin
Posts: 4886
Joined: 24-September 01
Member No.: 13



QUOTE (Sebastian Mares @ Aug 24 2011, 08:29) *
It appears to me that the low anchor was way too bad. Shouldn't the low anchor be at around the same quality as the contenders, but "slightly" worse than all of them?


Not sure about this one, I thought it should "calibrate the scale". (Because the overall quality is so high, it's less needed at the upper end)

If you don't use an anchor, what happens is that for a minor distortion users will tend to slam down the slider. The anchor serves as a reminder "what really bad really is".

It would be more useful if the anchor stayed the same throughout the tests, I guess. Probably the opportunity to test ffmpeg in one swoop was interesting. No idea if it was understood it is *this* bad.

FWIW, this is a somewhat relevant and interesting paper I hadn't seen before:
http://www.acourate.com/Download/BiasesInM...teningTests.pdf
Go to the top of the page
+Quote Post
Nezmer
post Aug 24 2011, 11:35
Post #4





Group: Members
Posts: 8
Joined: 6-May 11
Member No.: 90410



QUOTE (Garf @ Aug 24 2011, 10:35) *
Probably the opportunity to test ffmpeg in one swoop was interesting. No idea if it was understood it is *this* bad.


The AAC and Vorbis encoders in FFmpeg/libav were written to produce valid bitstreams without implementing any sophisticated optimisations. So, the results here shouldn't be a surprise.
Go to the top of the page
+Quote Post
Garf
post Aug 24 2011, 12:46
Post #5


Server Admin


Group: Admin
Posts: 4886
Joined: 24-September 01
Member No.: 13



QUOTE (Nezmer @ Aug 24 2011, 12:35) *
QUOTE (Garf @ Aug 24 2011, 10:35) *
Probably the opportunity to test ffmpeg in one swoop was interesting. No idea if it was understood it is *this* bad.


The AAC and Vorbis encoders in FFmpeg/libav were written to produce valid bitstreams without implementing any sophisticated optimisations. So, the results here shouldn't be a surprise.


Sorry, but this just isn't true for the ffmpeg AAC encoder. Have you actually looked at it? It's reasonably sophisticated, more sophisticated than FAAC for example. It has a real psymodel, 3 different quantization loop algorithms, proper short block switching, etc.

Even so, there's no particular reason to believe a non-sophisticated AAC encoder must terribly suck. Again, FAAC is good reference.
http://listeningtests.t35.me/html/AAC_at_1...est_results.htm

As far as I can tell, the problem is that it is utterly riddled with bugs and was probably never properly tested and debugged. It might be misdesigned too, but I feel like I'm sticking out my neck here because I could be wrong on that - maybe the current design works fine if you fix the bugs.

The ffmpeg AAC encoder is crap because it's buggy and insufficiently tested. Not because it's missing sophisticated algorithms.
Go to the top of the page
+Quote Post
Nezmer
post Aug 24 2011, 18:26
Post #6





Group: Members
Posts: 8
Joined: 6-May 11
Member No.: 90410



QUOTE (Garf @ Aug 24 2011, 13:46) *
QUOTE (Nezmer @ Aug 24 2011, 12:35) *
QUOTE (Garf @ Aug 24 2011, 10:35) *
Probably the opportunity to test ffmpeg in one swoop was interesting. No idea if it was understood it is *this* bad.


The AAC and Vorbis encoders in FFmpeg/libav were written to produce valid bitstreams without implementing any sophisticated optimisations. So, the results here shouldn't be a surprise.


Sorry, but this just isn't true for the ffmpeg AAC encoder. Have you actually looked at it? It's reasonably sophisticated, more sophisticated than FAAC for example. It has a real psymodel, 3 different quantization loop algorithms, proper short block switching, etc.

Even so, there's no particular reason to believe a non-sophisticated AAC encoder must terribly suck. Again, FAAC is good reference.
http://listeningtests.t35.me/html/AAC_at_1...est_results.htm

As far as I can tell, the problem is that it is utterly riddled with bugs and was probably never properly tested and debugged. It might be misdesigned too, but I feel like I'm sticking out my neck here because I could be wrong on that - maybe the current design works fine if you fix the bugs.

The ffmpeg AAC encoder is crap because it's buggy and insufficiently tested. Not because it's missing sophisticated algorithms.


I stand corrected.

The AAC encoder still needs `-strict experimental` to be enabled and I assumed they would distribue a basic encoder first then gradually implement optimisations.

Looking at the git log of 'aacenc.c', The last four commits contain three fixes and one library change. But before that, the psymodel seems to have been the focus of the work accomplished earlier this year.

How does all this affect the quality of the encoder? I don't know.
Go to the top of the page
+Quote Post

Posts in this topic
- IgorC   Public AAC Listening Test @ ~96 kbps [July 2011]: Results   Aug 23 2011, 19:56
- - benski   It would be interesting to do a rank-sum analysis ...   Aug 23 2011, 20:18
|- - Garf   QUOTE (benski @ Aug 23 2011, 21:18) It wo...   Aug 23 2011, 20:27
||- - benski   QUOTE (Garf @ Aug 23 2011, 15:27) QUOTE (...   Aug 23 2011, 20:54
||- - Garf   QUOTE (benski @ Aug 23 2011, 21:54) The F...   Aug 23 2011, 21:11
|- - C.R.Helmrich   QUOTE (benski @ Aug 23 2011, 21:18) ... w...   Aug 23 2011, 20:42
||- - Garf   QUOTE (C.R.Helmrich @ Aug 23 2011, 21:42)...   Aug 23 2011, 20:47
|||- - lvqcl   QUOTE (Garf @ Aug 23 2011, 23:47) Basical...   Aug 27 2011, 20:22
|||- - no404error   QUOTE (lvqcl @ Aug 27 2011, 22:22) CVBR, ...   Sep 4 2011, 04:01
||- - benski   QUOTE (C.R.Helmrich @ Aug 23 2011, 15:42)...   Aug 23 2011, 20:52
|- - IgorC   QUOTE (benski @ Aug 23 2011, 16:18) It wo...   Aug 23 2011, 21:01
|- - benski   QUOTE (IgorC @ Aug 23 2011, 16:01) QUOTE ...   Aug 23 2011, 21:07
|- - IgorC   QUOTE (benski @ Aug 23 2011, 17:07) Actua...   Aug 23 2011, 21:11
- - IgorC   I should also mention that I've participated i...   Aug 23 2011, 21:30
- - zima   Maybe there could be a legend for X-axis, the abbr...   Aug 23 2011, 21:37
- - lvqcl   It is interesting that QT tvbr and cvbr encoded fi...   Aug 23 2011, 22:05
- - IgorC   zima, will fix it later. QUOTE (lvqcl @ Au...   Aug 23 2011, 22:11
- - Alexxander   Thanks to all who participated in this test and to...   Aug 23 2011, 22:31
|- - Garf   QUOTE (Alexxander @ Aug 23 2011, 23:31) I...   Aug 23 2011, 23:02
- - Dakeryas   Many thanks for the test ! Interesting to not...   Aug 23 2011, 23:11
- - IgorC   I've noticed that previous version of Nero 1.0...   Aug 23 2011, 23:51
- - Gornot   To be perfectly honest, I am surprised that FhG di...   Aug 24 2011, 00:29
- - /mnt   Interesting results, I gotta see if the pre-echo h...   Aug 24 2011, 01:23
- - kennedyb4   If it is fair to say that many of the samples were...   Aug 24 2011, 01:42
- - Sebastian Mares   It appears to me that the low anchor was way too b...   Aug 24 2011, 07:29
|- - Garf   QUOTE (Sebastian Mares @ Aug 24 2011, 08...   Aug 24 2011, 09:35
|- - Nezmer   QUOTE (Garf @ Aug 24 2011, 10:35) Probabl...   Aug 24 2011, 11:35
|- - Garf   QUOTE (Nezmer @ Aug 24 2011, 12:35) QUOTE...   Aug 24 2011, 12:46
|- - Nezmer   QUOTE (Garf @ Aug 24 2011, 13:46) QUOTE (...   Aug 24 2011, 18:26
- - greynol   I was wondering the same thing.   Aug 24 2011, 07:49
- - C.R.Helmrich   Some bit-rate statistics which were presented in p...   Aug 24 2011, 18:27
|- - Garf   QUOTE (C.R.Helmrich @ Aug 24 2011, 19:27)...   Aug 25 2011, 07:29
- - Zarggg   Just looking for a quick verification on whether I...   Aug 25 2011, 18:06
|- - IgorC   QUOTE (Zarggg @ Aug 25 2011, 14:06) Am I ...   Aug 25 2011, 18:28
- - greynol   CVBR and TVBR are statistically tied. One did not...   Aug 25 2011, 18:12
|- - Zarggg   QUOTE (greynol @ Aug 25 2011, 13:12) CVBR...   Aug 25 2011, 22:47
|- - Garf   QUOTE (greynol @ Aug 25 2011, 19:12) CVBR...   Aug 26 2011, 07:20
- - greynol   That assumes facts not in evidence.   Aug 25 2011, 19:04
- - Gecko   First, thank you IgorC and everyone involved! ...   Aug 25 2011, 20:24
|- - IgorC   QUOTE (Gecko @ Aug 25 2011, 16:24) First,...   Aug 26 2011, 03:30
||- - mjb2006   QUOTE (IgorC @ Aug 25 2011, 20:30) I do n...   Aug 26 2011, 05:18
||- - IgorC   QUOTE (mjb2006 @ Aug 26 2011, 01:18) Anyw...   Aug 26 2011, 05:24
|- - Garf   QUOTE (Gecko @ Aug 25 2011, 21:24) a) Do ...   Aug 26 2011, 07:16
|- - IgorC   QUOTE (Garf @ Aug 26 2011, 03:16) Always ...   Aug 26 2011, 08:51
- - mjb2006   Even though I sent in results, they didn't get...   Aug 25 2011, 22:50
- - Gecko   Thank you IgorC and Garf for answering my question...   Aug 26 2011, 10:28
|- - IgorC   QUOTE (Gecko @ Aug 26 2011, 06:28) Given ...   Aug 26 2011, 11:20
||- - Garf   QUOTE (IgorC @ Aug 26 2011, 12:20) QUOTE ...   Aug 27 2011, 17:48
|- - Garf   QUOTE (Gecko @ Aug 26 2011, 11:28) In the...   Aug 27 2011, 17:39
- - IgorC   I found the first and the last graphs to be partic...   Aug 27 2011, 21:46
|- - lvqcl   QUOTE (IgorC @ Aug 28 2011, 00:46) I thin...   Aug 27 2011, 22:20
- - IgorC   Yeah   Aug 27 2011, 22:39
- - IgorC   BTW if someone want to organize the next public te...   Oct 1 2011, 20:21
- - jukkap   How about 48kbps HE AAC ? Or low bitrate multiform...   Oct 1 2011, 20:49
|- - IgorC   QUOTE (jukkap @ Oct 1 2011, 16:49) How ab...   Oct 10 2011, 23:17
|- - jukkap   QUOTE (IgorC @ Oct 10 2011, 23:17) The la...   Oct 20 2011, 10:04
- - IgorC   Well, the last time we have tested LC-AAC encoders...   Oct 1 2011, 21:58
- - jukkap   QUOTE (IgorC @ Oct 1 2011, 22:58) Can You...   Oct 2 2011, 05:10
- - IgorC   Late but still here. Some participants have answ...   Oct 23 2011, 07:14


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 19th December 2014 - 01:50