IPB

Welcome Guest ( Log In | Register )

Multiformat Listening Test @ 64 kbps - FINISHED
Sebastian Mares
post Aug 16 2007, 00:00
Post #1





Group: Members
Posts: 3629
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



The much awaited results of the Public, Multiformat Listening Test @ 64 kbps are ready - partially. So far, I only uploaded an overall plot along with a zoomed version. The details will be available tomorrow. You can also download the encryption key on the results page that is located here:

http://www.listening-tests.info/mf-64-1/results.htm
http://www.listening-tests.info/mf-64-1/resultsz.png

Nero and WMA Professional 10 are tied and WMA Professional 10 is tied to Vorbis. Vorbis however performed worse than Nero. Of course, High Anchor is best and Low Anchor loses.

This one goes to the experts: How would you rank codecs in such a situation, where A=B and B=C, but C<A?


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
 
Start new topic
Replies
guruboolez
post Aug 16 2007, 01:21
Post #2





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



My personal results:
CODE

WMAPro high Vorbis low HEAAC
2.3 3.7 2.0 1.0 3.2
2.0 3.0 1.5 1.0 2.5
2.0 2.5 2.5 1.0 1.7
2.8 4.3 3.2 1.5 3.8
2.5 4.5 2.8 1.0 1.8
2.7 2.5 2.0 1.0 1.5
1.8 5.0 1.5 1.0 3.0
1.8 3.5 3.0 1.0 2.2
2.0 3.5 3.0 1.0 2.3
3.5 3.0 3.0 1.0 2.0
2.0 3.0 2.0 1.0 1.7
1.5 2.3 1.3 1.0 1.5
4.0 3.0 3.5 1.0 4.0
3.5 3.0 2.5 1.0 2.8
2.1 1.5 3.0 1.0 2.0
3.0 4.5 2.0 1.5 3.0
1.2 3.5 2.0 1.0 1.5
3.5 3.0 2.0 1.0 2.0

FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Tukey HSD analysis

Number of listeners: 18
Critical significance: 0.05
Tukey's HSD: 0.574

Means:

high WMAPro Vorbis HEAAC low
3.29 2.46 2.38 2.36 1.06

-------------------------- Difference Matrix --------------------------

WMAPro Vorbis HEAAC low
high 0.839* 0.917* 0.933* 2.239*
WMAPro 0.078 0.094 1.400*
Vorbis 0.017 1.322*
HEAAC 1.306*
-----------------------------------------------------------------------

high is better than WMAPro, Vorbis, HEAAC, low
WMAPro is better than low
Vorbis is better than low
HEAAC is better than low


For the first time in listening tests my personal results are more evangelical than the collective one... no winner nor loser for my ears.

A direct comparison between my average scores and the collective one:

CODE
          collective   guruboolez  (diff)
low          1.55         1.06     -0.49
HE-AAC       3.74         2.36     -1.38
VORBIS       3.32         2.38     -0.94
WMAPRO       3.52         2.46     -1.06
high         4.59         3.29     -1.30
            ______       ______    ______
             3.34         2.31     -1.03

Compared to the whole group of testers my global evaluation for all competitors is clearly more harsh (-1.03 points on average), especially with the high anchor (-1.3 points) and HE-AAC (biggest deviation with -1.38 points). It confirms the lake of sympathy I feel for the SBR trick (there's several complains in my log files against the "SBR texture/noise"). I'm more disappointed by the high anchor which doesn't sound great to my ears. I expected more from LC-AAC two years after my previous test at 96 kbps.

WMAPro is a weird case. I'm not familar at all with this format (I never tested since its last metamorphosis in WMP11) and the new kind of distortion it produces. I disliked it on the beginning but I was much more enthousiastic after some times. Indeed, the second half of tested samples was better marked than the first one while it was at best the same for all other competitors. In other words my notation was more harsh during the second half but WMAPro's one has drastically grown in this severe period huh.gif
WMAPro artefacts were close to HE-AAC ones; it has a stronger smearing (cf kraftwerk, eig...) and share the same kind of SBRish issue (noise packets altering tonal sound, cymbals...) but often with less annoyance. It also has a kind of "noise sharpening" (for people knowing this foobar2000's plug-in) which tends to add some energy to high frequencies. Sound is often a bit brighter than reference to my ears. It's unexpected, not necessary a good thing but I find it rather pleasant in some situations, and certainly more enjoying than stereo reduction, pre-echo, lowpass or noise filtering. I simply fear that this kind of enhancement would quickly appear as tiresome (like noise sharpening IMO). That's why I wonder if I would still consider WMAPro so kindly with additionnal experience with this encoder and its own texture...

I was never fond of Vorbis at <80 kbps so I'm not surprised to see it inferior to HE-AAC with a confidence >95%. It often sound coarse, fat, with serious stereo issues (and a bit lowpassed too, but a smaller one would maybe increase the ringing...). I'm simply disappointed that for my taste no other format could currently outdistance this format.


As a consequence I'm disappointed. I maybe expected a miracle too soon after reading other people's comments. I will see in a future test if 80 or 96 kbps are more enjoyable for my taste.

This post has been edited by guruboolez: Aug 16 2007, 01:26
Go to the top of the page
+Quote Post

Posts in this topic
- Sebastian Mares   Multiformat Listening Test @ 64 kbps - FINISHED   Aug 16 2007, 00:00
- - guruboolez   Wow, thanks a lot for posting so fast these result...   Aug 16 2007, 00:06
|- - -Nepomuk-   Compare to the last 48kbit/s listening test, 64kbi...   Aug 16 2007, 00:27
||- - ff123   QUOTE (-Nepomuk- @ Aug 15 2007, 16...   Aug 16 2007, 01:43
||- - guruboolez   QUOTE (ff123 @ Aug 16 2007, 01:43) It...   Aug 16 2007, 01:52
|- - Sebastian Mares   QUOTE (guruboolez @ Aug 16 2007, 01:06) E...   Aug 16 2007, 00:32
- - rjamorim   Very interesting, Sebastian. Congratulations, and ...   Aug 16 2007, 00:28
- - kdo   Nice! I'm a little surprised that Vorbis ...   Aug 16 2007, 00:34
- - Sebastian Mares   All you need is Chunky! http://www.phong.org/c...   Aug 16 2007, 00:45
|- - kdo   QUOTE (Sebastian Mares @ Aug 16 2007, 01...   Aug 16 2007, 00:48
- - guruboolez   My personal results: CODE WMAPro high Vorbis low H...   Aug 16 2007, 01:21
- - kwanbis   QUOTE (Sebastian Mares @ Aug 15 2007, 23...   Aug 16 2007, 01:58
|- - echo   QUOTE (kwanbis @ Aug 16 2007, 03:58) QUOT...   Aug 17 2007, 00:36
|- - Woodinville   QUOTE (kwanbis @ Aug 15 2007, 17:58) QUOT...   Aug 21 2007, 20:48
- - kdo   Very interesting. After all my results are not so ...   Aug 16 2007, 02:00
- - kennedyb4   It seems that Itunes at 96 VBR has outscored Itune...   Aug 16 2007, 02:58
- - ff123   QUOTE (Sebastian Mares @ Aug 15 2007, 16...   Aug 16 2007, 05:42
- - vinnie97   Guru, my taste mirrors yours on Vorbis...anything ...   Aug 16 2007, 07:39
- - Slacker   Little Question: How do I use the key to see my re...   Aug 16 2007, 09:17
|- - kdo   QUOTE (Slacker @ Aug 16 2007, 10:17) Litt...   Aug 16 2007, 09:34
- - Alexxander   I used the key and decrypted results through java ...   Aug 16 2007, 09:46
|- - Sebastian Mares   QUOTE (Alexxander @ Aug 16 2007, 10:46) I...   Aug 16 2007, 12:49
- - muaddib   Wow, much more results than what I expected! T...   Aug 16 2007, 10:07
|- - muaddib   QUOTE (muaddib @ Aug 16 2007, 11:07) QUOT...   Sep 26 2007, 13:40
- - Alex B   Here are my personal results: CODE% Sample Averag...   Aug 16 2007, 11:53
- - thana   i downloaded the rar file and tried to process the...   Aug 16 2007, 14:12
|- - kdo   QUOTE (thana @ Aug 16 2007, 15:12) i down...   Aug 16 2007, 14:20
|- - Alex B   QUOTE (thana @ Aug 16 2007, 16:12) i down...   Aug 16 2007, 15:08
- - Rio   QUOTE (Sebastian Mares @ Aug 16 2007, 07...   Aug 16 2007, 15:11
|- - pdq   QUOTE (Rio @ Aug 16 2007, 10:11) QUOTE (S...   Aug 16 2007, 16:07
|- - benski   QUOTE (Rio @ Aug 16 2007, 10:11) QUOTE (S...   Aug 16 2007, 16:09
|- - ff123   QUOTE (benski @ Aug 16 2007, 08:09) One t...   Aug 16 2007, 16:20
|- - Whelkman   QUOTE (ff123 @ Aug 16 2007, 11:20) In the...   Aug 16 2007, 17:26
- - naylor83   Stupid question alert: If I ranked the reference,...   Aug 16 2007, 15:58
- - Sebastian Mares   The decrypted result files will then contain the r...   Aug 16 2007, 16:00
- - naylor83   Stupid question alert (again): I'm trying to ...   Aug 16 2007, 16:13
- - guruboolez   You can use MrQuestionMan, foobar2000 or several o...   Aug 16 2007, 16:17
|- - naylor83   QUOTE (guruboolez @ Aug 16 2007, 17:17) 1...   Aug 16 2007, 16:19
- - Sebastian Mares   Does anyone know how to make Excel to refer to the...   Aug 16 2007, 21:39
|- - mezenga   QUOTE (Sebastian Mares @ Aug 16 2007, 12...   Aug 17 2007, 00:17
- - Sebastian Mares   Uploaded the plots for each sample. The correspond...   Aug 16 2007, 23:31
- - ff123   Interesting. he-aac had some clear winners over w...   Aug 17 2007, 00:18
- - rockcake   I'd also like to give a big thankyou to Sebast...   Aug 17 2007, 05:32
|- - TechVsLife   QUOTE (rockcake @ Aug 17 2007, 00:32) I...   Aug 17 2007, 07:43
- - IgorC   Thanks for test. Nero has done a good work.   Aug 17 2007, 08:05
- - Alexxander   Congrat Nero! I can't believe WMA Pro 10 ...   Aug 17 2007, 09:30
|- - halb27   QUOTE (Alexxander @ Aug 17 2007, 10:30) ....   Aug 17 2007, 10:21
||- - Alexxander   QUOTE (halb27 @ Aug 17 2007, 11:21) ...Mo...   Aug 17 2007, 11:29
|- - muaddib   QUOTE (Alexxander @ Aug 17 2007, 10:30) C...   Aug 17 2007, 10:22
- - Ivan Dimkovic   CBR, in this context, means: "Fixed bit rate...   Aug 17 2007, 11:40
- - Alex B   Sebastian, Could you possibly post the average re...   Aug 17 2007, 12:45
|- - muaddib   QUOTE (Alex B @ Aug 17 2007, 13:45) Could...   Aug 17 2007, 13:05
|- - Alex B   QUOTE (muaddib @ Aug 17 2007, 15:05) It i...   Aug 17 2007, 13:37
- - muaddib   This looks much better now. Nero is at its place ...   Aug 17 2007, 14:02
- - lexor   QUOTE (Sebastian Mares @ Aug 15 2007, 20...   Aug 19 2007, 15:44
|- - halb27   QUOTE (lexor @ Aug 19 2007, 16:44) QUOTE ...   Aug 19 2007, 22:37
- - Jillian   How about use approximation instead of equality.   Aug 19 2007, 16:06
- - Nikaki   Interesting results. I only use Vorbis for high bi...   Aug 22 2007, 15:20
|- - TechVsLife   Wouldn't it make more sense just to go with on...   Aug 22 2007, 16:40
|- - Junon   QUOTE (TechVsLife @ Aug 22 2007, 17:40) W...   Aug 22 2007, 17:10
|- - TechVsLife   Thanks, I forgot about that whole dimension to the...   Aug 22 2007, 18:16
- - Woodinville   Out of curiousity, why is there no castinettes in ...   Aug 27 2007, 22:31
|- - [JAZ]   QUOTE (Woodinville @ Aug 27 2007, 23:31) ...   Aug 27 2007, 22:46
|- - Woodinville   QUOTE ' date='Aug 27 2007, 14:46' post...   Aug 28 2007, 18:03
- - Sebastian Mares   Well, I asked for sample suggestions long before t...   Aug 28 2007, 18:08
- - muaddib   How many of you that took test did use Headphones?...   Aug 31 2007, 15:28
|- - elmar3rd   QUOTE (muaddib @ Aug 31 2007, 14:28) How ...   Aug 31 2007, 15:54
||- - IgorC   Headphones for 17 samples. Loudspeakers for White...   Aug 31 2007, 17:19
|- - Alex B   I too used the Koss PortaPro headphones, which are...   Aug 31 2007, 20:08
- - ff123   headphones for me   Aug 31 2007, 15:35
- - kdo   good headphones (mid-price Beyerdynamic) with som...   Aug 31 2007, 17:44
- - naylor83   Headphones: Koss PortaPro   Aug 31 2007, 19:43
- - muaddib   It seems to me that in most cases headphones revea...   Sep 3 2007, 10:58
|- - naylor83   QUOTE (muaddib @ Sep 3 2007, 11:58) It se...   Sep 3 2007, 11:17
|- - muaddib   QUOTE (naylor83 @ Sep 3 2007, 12:17) Coul...   Sep 3 2007, 12:23
- - Whelkman   I used a pair of Sennheiser PX 100s for the test. ...   Sep 3 2007, 19:28
- - kdo   All of a sudden, I have got a small question -- ab...   Sep 6 2007, 13:54
- - Sebastian Mares   Within a sample plot, all bars should have the sam...   Sep 6 2007, 15:29
|- - kdo   QUOTE (Sebastian Mares @ Sep 6 2007, 16:2...   Sep 6 2007, 16:06
- - Sebastian Mares   Maybe someone with more knowledge in statistics ca...   Sep 6 2007, 17:20
- - robert   Who said all bars should be equal? What do you wan...   Sep 6 2007, 18:00
- - Sebastian Mares   In my results (and Roberto's, Guru's and f...   Sep 6 2007, 23:06
|- - naylor83   QUOTE (Sebastian Mares @ Sep 7 2007, 00:0...   Sep 6 2007, 23:11
- - ff123   For this type of analysis, the error bars are all ...   Sep 7 2007, 03:48
- - robert   So the bars do not represent the distribution of d...   Sep 7 2007, 10:58
|- - ff123   QUOTE (robert @ Sep 7 2007, 02:58) So the...   Sep 8 2007, 03:38
- - Moguta   I would've loved to see MP3 involved in this t...   Sep 8 2007, 02:56
- - Sebastian Mares   Yes, this is normal and depends on the mood, the l...   Sep 26 2007, 14:28
- - benwaggoner   Say, are there any plans for doing a new test here...   Apr 10 2008, 23:25
|- - benski   QUOTE (benwaggoner @ Apr 10 2008, 18:25) ...   Apr 10 2008, 23:29
|- - benwaggoner   QUOTE (benski @ Apr 10 2008, 14:29) QUOTE...   Apr 11 2008, 02:57
- - Sebastian Mares   The reason why WMA was tested in CBR mode is that ...   Apr 11 2008, 18:43
|- - benwaggoner   QUOTE (Sebastian Mares @ Apr 11 2008, 09...   Apr 11 2008, 21:20
- - Sebastian Mares   The test performed by NSTL featured WMA in CBR mod...   Apr 11 2008, 21:45
|- - benwaggoner   QUOTE (Sebastian Mares @ Apr 11 2008, 12...   Apr 11 2008, 23:28
- - hellokeith   QUOTE (benwaggoner @ Apr 11 2008, 15:20) ...   Apr 12 2008, 08:27
|- - benwaggoner   QUOTE (hellokeith @ Apr 11 2008, 23:27) N...   Apr 13 2008, 06:19
- - IgorC   I tried 1 and 2 pass CBR wma10 at 64 kbit/s in pas...   Apr 12 2008, 12:55
- - vinnie97   I'm still anxiously awaiting the forthcoming ~...   Apr 14 2008, 08:30


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 24th July 2014 - 03:20