IPB

Welcome Guest ( Log In | Register )

Alternative Multiformat Listening Test @ 128 kbps, at SoundExpert with the same contenders
Serge Smirnoff
post Jan 13 2006, 17:46
Post #1





Group: Members
Posts: 370
Joined: 14-December 01
Member No.: 641



As Sebastian Mares have practically finished accepting test results from participants I would like to offer for all who care to take part in alternative listening test with the same codec contenders but different sound samples. Testing methodology is also a bit different and makes testing much easier because sound artifacts are clearly audible in most cases. The main task of a listener is to grade annoyance of those artifacts.

Test files could be downloaded from here – ftp://www.soundexpert.info
Each time you click the link you’ll get random test file for one of these codecs:
• Nero AAC 3.1.0.2
• iTunes AAC 6.0.1.3
• LAME 3.97 Beta 2
• Ogg Vorbis AoTuV 4.51 Beta
• WMA Professional 9.1
• Shine 0.1.4 (Low Anchor)
Inside zip file you’ll find brief instruction and form for submitting results. The whole testing procedure is not hard and takes approx. 2-3 min., so you could easily test 5-10 files in one session. Each test file is 2-3 Mb.

Results of this test will appear on this page the same time with Sebastian Mares ones. After that they could be monitored in real time as new participants will add their grades.

Testing methodology used is described here - http://www.soundexpert.info/prozone.htm

Hope, comparison of both tests results will be interesting. If someone has any questions I am glad to answer.


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post
 
Start new topic
Replies
guruboolez
post Jan 15 2006, 04:28
Post #2





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



The testing procedure of this second listening test puzzled me.
I downloaded a file, and it appears that I can only rate one encoder for a given sample. It's a single A-B procedure. There's nothing wrong with that. What perplexed me is the role of the anchor. Isn't it intended to prevent the listener from temperamentic rating? It implies that the listener could access to the anchor while testing the other contenders. But in your testing procedure, you can only access to one encoded file and the hidden reference; the anchor, like all other contenders, are not accessible. The anchors can't therefore plays any role - at least not the anchor's one. It's just an additional contender.

It also mean that the listener can't rate all competitor in a same row. I can download:
- LAME => hearing a distortion => give it the note of 2/5
- then the ANCHOR => finding it awful => give the note of 1/5
- then download again LAME => hearing the same distortion as before => give the note of 4 because I would consider is as much better than the anchor quality I still have in mind.

It's clearly recommended to evaluate all encodings in a same raw, and to compare them each others before rating them all. It's like ABCHR softwares are working. Or at least to have the possibility to rank all encodings in a short amount of time. With SoundExpert's procedure, it looks impossible and the ranking could vary according to the testing mood, leading to uncoherent results.

This post has been edited by guruboolez: Jan 15 2006, 04:30
Go to the top of the page
+Quote Post
Serge Smirnoff
post Jan 15 2006, 11:37
Post #3





Group: Members
Posts: 370
Joined: 14-December 01
Member No.: 641



QUOTE (guruboolez @ Jan 15 2006, 06:28 AM)
What perplexed me is the role of the anchor. Isn't it intended to prevent the listener from temperamentic rating? It implies that the listener could access to the anchor while testing the other contenders. But in your testing procedure, you can only access to one encoded file and the hidden reference; the anchor, like all other contenders, are not accessible. The anchors can't therefore plays any role - at least not the anchor's one. It's just an additional contender...
It's clearly recommended to evaluate all encodings in a same raw, and to compare them each others before rating them all. It's like ABCHR softwares are working. Or at least to have the possibility to rank all encodings in a short amount of time. With SoundExpert's procedure, it looks impossible and the ranking could vary according to the testing mood, leading to uncoherent results.
*

Yes. The absence of low anchor really increases dispersion of results but it has to be compensated by broad participation of testers. Target audience of SoundExpert is completely unprepared and in most cases has no idea of what listening tests are. Instead of educate and train them (which are hard and thankless in real world) I decided to offer the listening procedure as simple as possible. It utilizes basic skills of an average listener – just “like” and “dislike” with a few intermediate states. As artifacts are clearly audible the influence of “temperamentic rating” is not high indeed. Each person has its own “inborn scale of annoyances” and in this case it’s better just use it but not build to this particular listening test.

Off course, it is a compromise between simplicity of procedure and scientific significance of its results. SoundExpert is highly experimental research project and up till now it shows that this compromise works. I think more fruitful discussion will be possible when raw stats are available.

And now I just ask for volunteers to download and grade a test item. Indeed it’s more like a fun than a listening test. And as you see the results are pretty close to Sebastian’s ones.


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post

Posts in this topic
- Serge Smirnoff   Alternative Multiformat Listening Test @ 128 kbps   Jan 13 2006, 17:46
- - jido   QUOTE (Serge Smirnoff @ Jan 13 2006, 08:46 AM...   Jan 14 2006, 11:26
|- - Serge Smirnoff   QUOTE (jido @ Jan 14 2006, 01:26 PM)So the te...   Jan 14 2006, 12:19
- - loophole   Artificial stimuli? As opposed to music? Isn't...   Jan 14 2006, 12:32
|- - Serge Smirnoff   QUOTE (loophole @ Jan 14 2006, 02:32 PM)Artif...   Jan 14 2006, 12:46
- - DigitalDictator   Interesting to check out the comparisons at other ...   Jan 14 2006, 12:58
|- - Serge Smirnoff   QUOTE (DigitalDictator @ Jan 14 2006, 02:58 P...   Jan 14 2006, 13:46
- - DigitalDictator   QUOTE BTW, are you sure that mp2 has to be worse t...   Jan 14 2006, 14:00
|- - Serge Smirnoff   QUOTE (DigitalDictator @ Jan 14 2006, 04:00 P...   Jan 14 2006, 14:34
|- - Garf   QUOTE (Serge Smirnoff @ Jan 14 2006, 03:34 PM...   Jan 14 2006, 14:52
|- - guruboolez   QUOTE (Garf @ Jan 14 2006, 02:52 PM)QUOTE (Se...   Jan 14 2006, 15:27
|- - Serge Smirnoff   QUOTE (Garf @ Jan 14 2006, 04:52 PM)QUOTE (Se...   Jan 16 2006, 23:29
|- - halb27   QUOTE (Serge Smirnoff @ Jan 17 2006, 12:29 AM...   Jan 16 2006, 23:52
|- - Serge Smirnoff   QUOTE (halb27 @ Jan 17 2006, 01:52 AM)[What a...   Jan 17 2006, 00:03
- - Garf   Note the parts "256kbps", "Often...   Jan 14 2006, 15:36
|- - guruboolez   QUOTE (Garf @ Jan 14 2006, 03:36 PM)Note the ...   Jan 14 2006, 15:46
- - Jan S.   As far as I recall Klemm actually considered to mo...   Jan 14 2006, 15:56
- - Garf   If you spend more bits on the right coefficients, ...   Jan 14 2006, 16:15
|- - guruboolez   QUOTE (Garf @ Jan 14 2006, 04:15 PM)That...   Jan 14 2006, 16:56
|- - halb27   QUOTE (guruboolez @ Jan 14 2006, 05:56 PM)......   Jan 16 2006, 21:52
- - Garf   The problem of PS and SBR is that they are paramet...   Jan 14 2006, 18:13
- - guruboolez   The testing procedure of this second listening tes...   Jan 15 2006, 04:28
|- - Serge Smirnoff   QUOTE (guruboolez @ Jan 15 2006, 06:28 AM)Wha...   Jan 15 2006, 11:37
- - guruboolez   I understand. The procedure is indeed very easy to...   Jan 15 2006, 12:07
|- - Serge Smirnoff   QUOTE (guruboolez @ Jan 15 2006, 02:07 PM)The...   Jan 15 2006, 12:49
- - Sebastian Mares   Just noticed something on the page... QUOTE aac A...   Jan 15 2006, 13:15
|- - Serge Smirnoff   QUOTE (Sebastian Mares @ Jan 15 2006, 03:15 P...   Jan 15 2006, 18:25
- - Ivan Dimkovic   QUOTE So, using layers at higher than sweet spot b...   Jan 16 2006, 23:43
- - Ivan Dimkovic   "Sweet spot" comes from the relatively s...   Jan 17 2006, 00:11
- - Sagittaire   Just my 2 cents ... ;-) 1) Well ... not really 12...   Jan 18 2006, 13:26
- - Gabriel   QUOTE 1) Well ... not really 128 Kbps test but 140...   Jan 18 2006, 13:39
|- - Sagittaire   QUOTE The resulting overall on overall music is ab...   Jan 18 2006, 14:15
- - Alex B   The bitrates were measured with a big amount of co...   Jan 18 2006, 13:45
- - Alex B   BTW, this thread is about the alternative Sound Ex...   Jan 18 2006, 13:58
- - Garf   The ability of a codec to distribute more bits to ...   Jan 18 2006, 14:28
- - pepoluan   Just noticed that on SoundExpert site, the lower b...   Jan 19 2006, 18:01
- - Serge Smirnoff   QUOTE (pepoluan @ Jan 19 2006, 08:01 PM)Consi...   Jan 20 2006, 00:00


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 3rd September 2014 - 09:35