IPB

Welcome Guest ( Log In | Register )

Public MP3 Listening Test @ 128 kbps - FINISHED
Sebastian Mares
post Nov 24 2008, 22:30
Post #1





Group: Members
Posts: 3637
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



The much awaited results of the Public, MP3 Listening Test @ 128 kbps are ready - partially. So far, I only uploaded an overall plot along with a zoomed version. The details will be available in the next few days. You can also download the encryption key along with the submitted results on the results page that is located here: http://www.listening-tests.info/mp3-128-1/results.htm

The results show that all encoders are tied on first place, except l3enc which of course comes out last being the low anchor.

What is interesting to see is how the MP3 codec actually evolved since its first days (l3enc was the first MP3 software encoder back in 1994 when it was released) and how it is still competitive with newer formats like AAC or Ogg Vorbis.

Another very interesting thing, which was also one of the goals for this test, is that Fraunhofer and especially Helix, which both outperform LAME in terms of encoding speed, are still very competitive. While statistically being tied to LAME on first place, Helix actually even received a higher rating than LAME 3.98.2 - and this at 90x encoding speed! Even FhG received a slightly higher score at least against LAME 3.97 which was the recommended encoder by the Hydrogenaudio community for a long time. But again, statistically, they are all tied so there is no quality winner.



The quality at 128 kbps is very good and MP3 encoders improved a lot since the last test. This was the last test conducted by me at this bitrate. It's time to move to bitrates like 96 kbps or 80 kbps.

Here is a zoomed version of the plot showing the competitors only and leaving out the low anchor l3enc.



Finally, I would like to thank everyone who participated!

EDIT: Whoops, the link to the results was pointing to the 64 kbps multiformat test by mistake. Corrected now.

This post has been edited by Sebastian Mares: Nov 24 2008, 22:46


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
 
Start new topic
Replies
IgorC
post Nov 25 2008, 17:23
Post #2





Group: Members
Posts: 1580
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



I can confirm statements of /mnt . Helix isn't good on my collection of rock and metal samples. But it's good for a bunch of overkill samples.
I noticed that LAME has a problem with a few first seconds of each sample while Helix doesn't. It can explain surprising reults at least partially.

Should admit interesing test. Thank you, Sebastian.
Go to the top of the page
+Quote Post
ff123
post Nov 25 2008, 17:26
Post #3


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



QUOTE (IgorC @ Nov 25 2008, 08:23) *
I can confirm statements of /mnt . Helix isn't good on my collection of rock and metal samples. But it's good for a bunch of overkill samples.
I noticed that LAME has a problem with a few first seconds of each sample while Helix doesn't. It can explain surprising reults at least partially.

Should admit interesing test. Thank you, Sebastian.


There should have been an option in the configuration files to specify that the first second or two of each sample would be ignored. This would prevent the problem you mention creeping in.
Go to the top of the page
+Quote Post
guruboolez
post Nov 25 2008, 18:04
Post #4





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



The 2 or 3 first seconds were already ignored in this test.

Interesting results anyway. Conclusion is far from what I reached in the past. I only tested the last 11 samples ; my results are therefore not totally comparable but are significantly different:

iTunes: 2.98
Lame 3.98: 3.30
l3enc: 1.171.00
fraunhofer: 3.51
LAME 3.97: 3.68
Helix: 2.95

This is also the very first test I performed with my new headphone I just owned the day before I started the test. The new sound signature was so different and therefore so disturbing that I didn't bother to spend more than a few minutes to test and give an evaluation to each sample. It was a strange experience for me. I wonder how much a different headphone may change results. But it becomes clear to me that a different material configuration could heavily disturb a listener.

Anyway, even in this highly confused listening environment my results in this test tend to confirm that Helix doesn't please me at all, even with completely different samples / musical genre.

This post has been edited by guruboolez: Nov 25 2008, 18:18
Go to the top of the page
+Quote Post
TechVsLife
post Nov 25 2008, 21:44
Post #5





Group: Members
Posts: 195
Joined: 29-May 07
Member No.: 43837



@sebastian mares: thanks for the test!

@guruboolez

Helix doesn't please me at all

Your personal quick test shows (if not a typo):

helix:lame398::lame398:lame397. So acc to this, if helix is worse than lame 3.98, then for you lame 3.98 is just as much worse than 3.97. How would you explain?

I take it that it's impossible for any individual's result in the general test to be (statistically) meaningless, because it's repeated and blinded etc. (so it's never a "fluke"). So I'm trying to explain what that means in the context of a statistical tie for the group, esp. in those individual cases, like guruboolez, where there is NOT a tie.

1. all encoders are so close, that individual sensitivities/variances (or quirks, depending on your view of their significance) dominate more, even (or especially) in a group of more sensitive than average listeners. [this could sometimes result in ties for just one listener, if we are not talking about a specific weakness of an encoder repeated across several selections of music but very fine and specific differences, limited to specific sounds or instruments or genres, see #2]

2. the division by music genre (or instruments used etc.) seems important. is there a way to know if there is a division in the results this way, i.e. producing something other than a tie for the whole result set? [this could be true along with #1]. (I esp. care about classical.)

3. is there a way to know whether and to what extent an unduly low anchor masked or could mask substantial quality differences?

p.s. The sample by sample discussion is good and does address #2. The graph by sample is helpful--wonder if there is enough data to make informed statistically sound judgments by music type?)


QUOTE (guruboolez @ Nov 25 2008, 12:04) *
The 2 or 3 first seconds were already ignored in this test.

Interesting results anyway. Conclusion is far from what I reached in the past. I only tested the last 11 samples ; my results are therefore not totally comparable but are significantly different:

iTunes: 2.98
Lame 3.98: 3.30
l3enc: 1.171.00
fraunhofer: 3.51
LAME 3.97: 3.68
Helix: 2.95

This is also the very first test I performed with my new headphone I just owned the day before I started the test. The new sound signature was so different and therefore so disturbing that I didn't bother to spend more than a few minutes to test and give an evaluation to each sample. It was a strange experience for me. I wonder how much a different headphone may change results. But it becomes clear to me that a different material configuration could heavily disturb a listener.

Anyway, even in this highly confused listening environment my results in this test tend to confirm that Helix doesn't please me at all, even with completely different samples / musical genre.


This post has been edited by TechVsLife: Nov 25 2008, 21:55
Go to the top of the page
+Quote Post

Posts in this topic
- Sebastian Mares   Public MP3 Listening Test @ 128 kbps - FINISHED   Nov 24 2008, 22:30
- - /mnt   Wow am really shocked, Helix (Xing) has performed ...   Nov 24 2008, 22:52
- - Sebastian Mares   I kept telling you guys that the results will be q...   Nov 24 2008, 22:56
- - Neasden   Does that make Helix the new recommended MP3 encod...   Nov 24 2008, 23:00
|- - /mnt   QUOTE (Neasden @ Nov 24 2008, 23:00) Does...   Nov 24 2008, 23:05
- - Neasden   Yes it is, I just noticed it! I can't bel...   Nov 24 2008, 23:14
- - greynol   I don't think open source has anything to do w...   Nov 24 2008, 23:14
- - Sebastian Mares   If you submitted results, I recommend you look at ...   Nov 24 2008, 23:14
- - halb27   I am curious about the detailed results as my inte...   Nov 24 2008, 23:22
- - Neasden   They are all techincally tied, but Helix outperfor...   Nov 24 2008, 23:25
|- - halb27   QUOTE (Neasden @ Nov 25 2008, 00:25) ... ...   Nov 25 2008, 00:43
|- - Pio2001   Excuse me, but what is the correspondance between ...   Nov 25 2008, 00:48
|- - lvqcl   QUOTE (Pio2001 @ Nov 25 2008, 02:48) Excu...   Nov 25 2008, 01:16
- - Canar   Thank you very much Sebastian. We have some things...   Nov 24 2008, 23:51
- - DigitalDictator   This is indeed surprising. I'm sure I've s...   Nov 24 2008, 23:56
|- - halb27   QUOTE (DigitalDictator @ Nov 25 2008, 00...   Nov 25 2008, 00:11
||- - Pio2001   AAAAAAAAAAAARRRRRRRRGGGGHGHHHHHH !! They...   Nov 25 2008, 00:34
|- - melomaniac   I analyzed my results and the ranking of the encod...   Nov 25 2008, 09:19
|- - Pio2001   QUOTE (melomaniac @ Nov 25 2008, 09:19) A...   Nov 25 2008, 14:04
- - ZinCh   This is only 128k tests, so Helix in the winner in...   Nov 24 2008, 23:59
- - Neasden   I encoded a few tracks using -V150 (VBR range 0-15...   Nov 25 2008, 00:07
- - greynol   >Helix can be recomended for 128k encoding. Ho...   Nov 25 2008, 00:10
- - kwanbis   Wow (even if the difference between LAME 3.98.2 an...   Nov 25 2008, 00:42
- - krabapple   Sorry, it's not clear to me how many subjects ...   Nov 25 2008, 01:02
- - Pio2001   Thanks, I analyzed my own results. That's what...   Nov 25 2008, 01:37
- - Raiden   QUOTE (Pio2001 @ Nov 25 2008, 01:34) Oh, ...   Nov 25 2008, 02:12
|- - Canar   QUOTE (Raiden @ Nov 24 2008, 17:12) agree...   Nov 25 2008, 02:32
- - sld   Regarding statistics... the confidence intervals w...   Nov 25 2008, 04:51
|- - Sebastian Mares   QUOTE (sld @ Nov 25 2008, 04:51) Regardin...   Nov 25 2008, 09:02
- - Squeller   Is this claim correct? There has been no improveme...   Nov 25 2008, 08:46
- - halb27   Zoomed view is formally correct, but has a tendenc...   Nov 25 2008, 09:19
|- - Squeller   QUOTE (halb27 @ Nov 25 2008, 10:19) Zoome...   Nov 25 2008, 09:28
|- - halb27   QUOTE (Squeller @ Nov 25 2008, 10:28) Bas...   Nov 25 2008, 09:44
|- - melomaniac   QUOTE (Squeller @ Nov 25 2008, 09:28) Abo...   Nov 25 2008, 09:46
|- - sld   QUOTE (Squeller @ Nov 25 2008, 16:28) Bas...   Nov 25 2008, 20:05
- - memomai   Just confused. Helix worse than lame, Helix better...   Nov 25 2008, 10:08
|- - halb27   QUOTE (memomai @ Nov 25 2008, 11:08) Just...   Nov 25 2008, 10:30
|- - Alexxander   QUOTE (halb27 @ Nov 25 2008, 10:30) ... W...   Nov 25 2008, 10:55
|- - halb27   QUOTE (Alexxander @ Nov 25 2008, 11:55) ....   Nov 25 2008, 13:04
- - Alexxander   Before anything I have to thank Sebastian again fo...   Nov 25 2008, 10:44
- - muaddib   It is not good to conclude, from the results of th...   Nov 25 2008, 11:15
|- - Alexxander   QUOTE (muaddib @ Nov 25 2008, 11:15) It i...   Nov 25 2008, 12:37
- - Jan S.   Wouldn't it be possible to compare the varianc...   Nov 25 2008, 12:33
|- - Sebastian Mares   QUOTE (Jan S. @ Nov 25 2008, 12:33) Would...   Nov 25 2008, 12:44
- - robert   I would be more interested in Quartile, instead of...   Nov 25 2008, 13:01
|- - kwanbis   QUOTE (robert @ Nov 25 2008, 12:01) I wou...   Nov 25 2008, 13:06
|- - Sebastian Mares   QUOTE (robert @ Nov 25 2008, 13:01) I wou...   Nov 25 2008, 14:47
- - westgroveg   If anything the test shows samples where LAME need...   Nov 25 2008, 13:55
|- - Alex B   QUOTE (westgroveg @ Nov 25 2008, 14:55) I...   Nov 25 2008, 14:51
|- - uart   QUOTE (Alex B @ Nov 25 2008, 05:51) In ge...   Nov 25 2008, 16:30
- - /mnt   Just try some Metal tracks on Helix at V60, I guar...   Nov 25 2008, 15:19
- - Neasden   /mnt told me that Helix is not gapless, which is t...   Nov 25 2008, 16:07
|- - Synthetic Soul   QUOTE (Neasden @ Nov 25 2008, 15:07) /mnt...   Nov 25 2008, 17:21
|- - Alexxander   QUOTE (Synthetic Soul @ Nov 25 2008, 17:2...   Nov 25 2008, 18:27
- - DigitalDictator   Why would Helix struggle with metal? IIRC it also ...   Nov 25 2008, 16:15
|- - /mnt   QUOTE (DigitalDictator @ Nov 25 2008, 16...   Nov 25 2008, 16:38
- - Sebastian Mares   Well, one thing you have to consider is that Helix...   Nov 25 2008, 16:32
- - IgorC   I can confirm statements of /mnt . Helix isn't...   Nov 25 2008, 17:23
|- - ff123   QUOTE (IgorC @ Nov 25 2008, 08:23) I can ...   Nov 25 2008, 17:26
|- - Alex B   QUOTE (Sebastian Mares @ Nov 25 2008, 17...   Nov 25 2008, 17:34
|- - guruboolez   The 2 or 3 first seconds were already ignored in t...   Nov 25 2008, 18:04
|- - TechVsLife   @sebastian mares: thanks for the test! @gurub...   Nov 25 2008, 21:44
- - Sebastian Mares   QUOTE (ff123 @ Nov 25 2008, 17:26) QUOTE ...   Nov 25 2008, 18:08
|- - Alex B   QUOTE (Sebastian Mares @ Nov 25 2008, 19...   Nov 25 2008, 18:50
- - Jillian   I like the part where test result (quality and enc...   Nov 25 2008, 19:13
|- - halb27   QUOTE (Jillian @ Nov 25 2008, 20:13) I li...   Nov 25 2008, 21:13
- - Canar   QUOTE (Neasden @ Nov 25 2008, 07:07) /mnt...   Nov 25 2008, 19:34
- - Alex B   QUOTE (Synthetic Soul @ Nov 25 2008, 18:2...   Nov 25 2008, 19:36
- - Canar   Gapless encoding using Helix is possible by encodi...   Nov 25 2008, 19:51
- - Neasden   Do you mean having to "split" manually t...   Nov 25 2008, 20:06
|- - Alex B   QUOTE (Neasden @ Nov 25 2008, 21:06) Do y...   Nov 25 2008, 20:33
|- - guruboolez   QUOTE (Alex B @ Nov 25 2008, 21:33) Perso...   Nov 25 2008, 21:00
|- - Synthetic Soul   QUOTE (guruboolez @ Nov 25 2008, 20:00) T...   Nov 25 2008, 21:07
- - Sebastian Mares   Created 7 / 14 sample graphs... BTW, Tukey's ...   Nov 25 2008, 20:10
- - kwanbis   Sebastian, where it says, "The results are gr...   Nov 25 2008, 20:18
- - Bodhi   Great Job once again. Thank you Sebastian!   Nov 25 2008, 20:44
- - Neasden   QUOTE As Canar said, it's a hack. That alone ...   Nov 25 2008, 20:58
- - Neasden   unless someone patches Helix and release it with t...   Nov 25 2008, 21:05
- - Sebastian Mares   In case you are interested, here is a quick and di...   Nov 25 2008, 21:10
|- - halb27   QUOTE (Sebastian Mares @ Nov 25 2008, 22...   Nov 25 2008, 21:26
|- - benski   QUOTE (Sebastian Mares @ Nov 25 2008, 15...   Nov 25 2008, 21:50
- - Canar   The point I was trying to make was that though Hel...   Nov 25 2008, 21:14
- - Neasden   in this graph, LAME 3.98.2 seems the more stable e...   Nov 25 2008, 21:42
|- - Alex B   QUOTE (Neasden @ Nov 25 2008, 22:42) in t...   Nov 25 2008, 22:06
- - Sebastian Mares   The graphs for all samples are available on the re...   Nov 25 2008, 22:18
|- - halb27   QUOTE (Sebastian Mares @ Nov 25 2008, 23...   Nov 25 2008, 22:25
- - singaiya   Is anybody else not surprised that each contender ...   Nov 25 2008, 22:21
- - Sunhillow   Thank you for this great checkup, Sebastian! I...   Nov 25 2008, 22:35
- - Zilog Jones   I, as many others was also very surprised by the r...   Nov 25 2008, 22:53
|- - Sebastian Mares   QUOTE (Zilog Jones @ Nov 25 2008, 22:53) ...   Nov 25 2008, 23:24
- - sizetwo   Without adding fuel to the fire, I think its stran...   Nov 25 2008, 23:16
|- - sld   QUOTE (sizetwo @ Nov 26 2008, 06:16) Does...   Nov 26 2008, 05:14
|- - singaiya   QUOTE (sld @ Nov 25 2008, 20:14) You shou...   Nov 26 2008, 05:46
- - DigitalDictator   I've been asking this a couple of times, but I...   Nov 25 2008, 23:30
- - sizetwo   QUOTE Statistically, for the people who tested and...   Nov 25 2008, 23:34
|- - guruboolez   QUOTE (sizetwo @ Nov 26 2008, 00:34) Sorr...   Nov 25 2008, 23:52
|- - kwanbis   QUOTE (guruboolez @ Nov 25 2008, 22:52) N...   Nov 26 2008, 00:36
- - /mnt   I have posted some ABX logs and samples of tracks ...   Nov 26 2008, 00:49
- - JasonQ   Good test. Good to see that Helix had a solid sho...   Nov 26 2008, 06:18
- - sizetwo   QUOTE (guruboolez @ Nov 25 2008, 15:52) Q...   Nov 26 2008, 07:18
3 Pages V   1 2 3 >


Reply to this topicStart new topic
2 User(s) are reading this topic (2 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 21st December 2014 - 15:28