IPB

Welcome Guest ( Log In | Register )

General discussion of future public test
IgorC
post Dec 21 2011, 16:28
Post #1





Group: Members
Posts: 1540
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Hello,
Now when its sure that Apple encoder represents AAC standard very well the next step will be a public multiformat test.
No particular date or plan. Its a general discussion with the purpose to hear the opinions, points of view, suggestions.

Items to talk about (these numbers are first approximations):
1.The number of codecs: 4+low anchor. 4 is affordable number for one test.
2.The selection of codecs (basing on curret previous votes):
http://listening-test.blogspot.com/2011/06...reparation.html

Apple LC-AAC - the most optimal AAC encoder. (Well, though FhG was quite good too).
MP3 LAME 10 votes
Vorbis AoTuV 8.5 votes.
Opus/CELT 6 votes. (still no info when there will be Opus 1.0)

Its possible to change your votes.

(?) USAC (working name of future AACs successor ). No information about future availability of it.
(?) low anchor
(?) Settings, versions, etc.

3. Bitrate:
VBR 96 kbps. The last AAC public test was at 96 kbps (well, it was actually 100 kbps).
Also I think it will be more interesting to compare MP3 128-135 kbps and AAC/AoTuV 96-100 kbps. Probably a lot of people are interesting to trade off between compability/compression efficiency smile.gif

4. Samples.
Last time we have applied the technic of random selection of the samples. 20 samples.

The developers are very welcome to participate in this topic.

This test should be more interesting as it will include interesting (for use) codecs. biggrin.gif
Go to the top of the page
+Quote Post
 
Start new topic
Replies
IgorC
post Dec 25 2011, 18:18
Post #2





Group: Members
Posts: 1540
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Some feedback was received on #Hydrogenaudio and I would like to bring some information and answers basing my opinion on past public tests.


1. Can we go for higher bitrate (>96 kbps)?
There were 531 results for 64 kbps test and only 280 results for 96 kbps AAC test while it was open during more time. 280 results were just fairly enough to close the test.
64 kbps test has continued during 20 days, 96 kbps – 35 days (hence more difficult).
It’s not recomended to keep the test open for more than one month or so.

So it’s hard to imagine a public test at higher bitrate.


2.a) Will be it more reasonable don’t discard all results of a listener who submit more invalid results that it’s allowed by rules? Keep “good” results and discard only erroneous results?
First of all it worth to mention that this question isn’t new.
The problem is that it’s impossible to know if it was a really “good” result or it was a “lucky guess”. So it’s not accepting the “good” results and discarding the “bad” results but rather accepting “lucky guessing” results (which is still invalid one) and discarding “not lucky guessing”.

Answer: it’s more correct to discard all results of the listener who has past the limit of the invalid results (submit too much of them). Also see (1)

2b) One would say then we can check if the results of this one particular listener correlates enough well with the results of other listener then it’s ok to accept only good results .
The problem that later developers and memebers will complain why the rules weren’t applied homogeneously (to all listeners in the same manner).
Then the simple and effective decesion was made:
The rules were _strictly_ applied (as of previous public tests 64 amd 96 kbps). No context. You follow the rules – your results are in. If not – then no but you can start from zero. Simply as that.



Also my observation about placebo effect and the interpretation of the results during the public listening test.

The score 4.0 means “perceptible but not annoying”. And how should the results of the last AAC test be interpreted in case of 4.1 4.2 4.3 etc...?
2-3 steps (0.2-0.3) is considering like a “very few ones” (in general interpretation. It’s psychology so I won't opine here smile.gif ). But more than 2-3 steps arent “very few” . So one could say that 4.2-4.3 is a limit score where the high quality begins.
I dont know if its coincidence but this score (4.2-4.3) is also where placebo effect starts to reveal itself more. (1) Another interesting observation is that placebo leads to lower score than normal. There were quite enough results where the listener has put lower score if it was actually placebo. It leads to “flipped results” . It means if average results were Codec A > B > C > D then flipped ones (with placebo effect) will be D>C>B>A.

(oh lord, my english unsure.gif)


Speaking of this test we can go for both MP3 96 and 128 kbps. It's a period to throw the ideas.

This post has been edited by IgorC: Dec 25 2011, 18:28
Go to the top of the page
+Quote Post

Posts in this topic
- IgorC   General discussion of future public test   Dec 21 2011, 16:28
- - Steve Forte Rio   Nice... QUOTE Its possible to change your votes....   Dec 22 2011, 16:45
- - IgorC   Good. Guys, propose the codecs you are intereste...   Dec 22 2011, 17:03
- - IgorC   There is a first issue if MP3 will be tested at 96...   Dec 22 2011, 20:17
|- - NullC   QUOTE (IgorC @ Dec 22 2011, 12:17) There ...   Dec 22 2011, 21:44
|- - Nick.C   QUOTE (IgorC @ Dec 22 2011, 19:17) Possib...   Dec 22 2011, 22:34
- - lvqcl   8.5 votes? then: +1 for AAC +1 for Vorbis +0.5...   Dec 22 2011, 20:37
- - Kohlrabi   I'd like to see how Opus will perform here, so...   Dec 22 2011, 21:10
- - IgorC   Not all cards support automatic switching between ...   Dec 23 2011, 14:16
- - halb27   The fact that Lame resamples to 32 kHz at 96 kbps ...   Dec 23 2011, 16:37
|- - nu774   QUOTE (halb27 @ Dec 24 2011, 00:37) The f...   Dec 23 2011, 17:24
- - IgorC   Also this time I will be glad to be just a listene...   Dec 24 2011, 00:53
- - IgorC   Some feedback was received on #Hydrogenaudio and ...   Dec 25 2011, 18:18
- - Yaztromo   +1 QT AAC +1 Vorbis (Aotuv b603?) +1 LAME 3.99 +0....   Dec 26 2011, 02:33
- - Gecko   +1 for keeping the number of codecs small. Conside...   Dec 26 2011, 12:43
- - IgorC   Very low quality low anchor permits to analyse the...   Dec 26 2011, 14:10
|- - knik   How about FAAC, I think it could be good as anchor...   Mar 2 2012, 10:44
- - Kohlrabi   Any chance to see Musepack competing against curre...   Mar 2 2012, 12:41
- - jensend   USAC has been "released" with the confus...   Mar 3 2012, 08:06
|- - knik   QUOTE Anybody who thinks that MP3 should only be t...   Mar 3 2012, 09:50
- - Pretentiousman   +1 Apple LC-AAC +1 MP3 LAME @ 128-135 kbps +1 Vorb...   Mar 7 2012, 22:03
- - Gainless   I think it would be a good idea to make another AA...   Aug 9 2012, 14:14
- - Garf   QUOTE (Gainless @ Aug 9 2012, 15:14) I th...   Aug 9 2012, 16:09
|- - jensend   QUOTE (Garf @ Aug 9 2012, 09:09) Ideally ...   Aug 10 2012, 04:09
- - IgorC   QUOTE (Gainless @ Aug 9 2012, 10:14) I th...   Aug 16 2012, 14:48


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 29th July 2014 - 08:25