IPB

Welcome Guest ( Log In | Register )

15 Pages V  « < 13 14 15  
Closed TopicStart new topic
MP3 Listening Test at 128 kbps, Call for encoders and settings
Alex B
post Oct 9 2008, 11:05
Post #351





Group: Members
Posts: 1303
Joined: 14-September 05
From: Helsinki, Finland
Member No.: 24472



While we wait the test to begin it might be useful to revisit the comments that were posted in the 64kbps multiformat test's announce thread: http://www.hydrogenaudio.org/forums/index....showtopic=56397

In that thread I made some suggestions about how the test presentation and instructions could be developed further: http://www.hydrogenaudio.org/forums/index....st&p=509971

EDIT

In addition, the comments in the post-test thread make a good read:
http://www.hydrogenaudio.org/forums/index....showtopic=56851

This post has been edited by Alex B: Oct 9 2008, 11:28


--------------------
http://listening-tests.freetzi.com
Go to the top of the page
+Quote Post
Sebastian Mares
post Oct 11 2008, 09:08
Post #352





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



I am afraid that a major "overhaul" of the listening test page and the instruction is not going to be ready in time for this test. I am planning to change the listening test pages and make them somewhat more attractive, but 1. I am not really talented in designing so I have to see who can help me and 2. this is going to take some time so it will be only available for the next teast at earliest. I hope to have less ranked references in this test. Most of the people who perform listening tests are HA readers anyways so I was actually assuming that they know perform a "correct" test.


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
Alex B
post Oct 11 2008, 11:41
Post #353





Group: Members
Posts: 1303
Joined: 14-September 05
From: Helsinki, Finland
Member No.: 24472



I didn't want to put more pressure on you. smile.gif

I merely wanted to keep this discussion alive and posted the links for newcomers so that they can get a better impression of the complete testing process.


--------------------
http://listening-tests.freetzi.com
Go to the top of the page
+Quote Post
Sebastian Mares
post Oct 13 2008, 08:31
Post #354





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Small status update: I am trying to get in touch with schnofler for some changes in ABC/HR. Everything is pretty much done except for the ABC/HR configuration files and is ready to be uploaded.


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
Sebastian Mares
post Oct 15 2008, 22:44
Post #355





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Here is the updated bitrate table that also contains the encoding speeds (except for iTunes that has to be tested once the fixed version is out).
I didn't want to redo all tests on the single-core machine and comparing iTunes' single-core results to the dual-core results of the other encoders is not fair (at least if any of the other encoders makes use of multi-threading).

CODE
Sample 1 Sample 2 Sample 3 Sample 4 Sample 5 Sample 6 Sample 7 Sample 8 Sample 9 Sample 10 Sample 11 Sample 12 Sample 13 Sample 14 Max Min Avg Speed 1)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
LAME 3.97 | 97 126 138 149 146 150 95 147 109 158 149 194 133 159 194 95 139 18x
LAME 3.98.2 | 107 149 143 140 138 143 109 136 118 152 145 214 146 156 214 107 143 27x
FhG | 119 121 139 140 144 149 134 128 129 147 134 212 151 163 212 119 144 45x
iTunes | 118 117 139 145 133 125 141 120 126 148 151 192 159 158 192 117 141 N/A 2)
Helix | 114 110 126 151 149 151 131 138 97 152 117 228 143 173 228 97 141 90x
Low Anchor | 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 1,63x
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Max | 119 149 143 151 149 151 141 147 129 158 151 228 159 173
Min | 97 110 126 140 133 125 95 120 97 147 117 192 133 156
Avg | 111 125 137 145 142 144 122 134 116 151 139 208 146 162 142
Max - Min | 22 39 17 11 16 26 46 27 32 11 34 36 26 17


1) These results are only an approximate value and were obtained on a dual-core Intel E6550 with 2 GB RAM running on Windows Vista SP1 32-bit
2) Encoding speed will be measured with iTunes once the bug-fixed version was released. The buggy version encoded at 18x but produced lower bitrates.


Edit 1: IMPORTANT: Please notice that the low anchor wasn't taken into consideration for building the min, max, delta and average bitrates.
Edit 2: Sorry for breaking the layout - I used a codebox so I don't exactly know what is going wrong here. sad.gif

This post has been edited by Sebastian Mares: Oct 15 2008, 22:47


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
Polar
post Oct 16 2008, 10:50
Post #356





Group: Members
Posts: 266
Joined: 12-February 04
Member No.: 11970



Sorry to flutter the dovecotes perhaps, but wouldn't it be fairer to rename the test into 140k then, in stead of 128k?
Go to the top of the page
+Quote Post
Alex B
post Oct 16 2008, 11:38
Post #357





Group: Members
Posts: 1303
Joined: 14-September 05
From: Helsinki, Finland
Member No.: 24472



QUOTE (Polar @ Oct 16 2008, 12:50) *
Sorry to flutter the dovecotes perhaps, but wouldn't it be fairer to rename the test into 140k then, in stead of 128k?

Short test samples do not represent the average bitrate behavior of complete tracks of various genres in a media library. In my tests the overall average was 127.8 kbps. I think it is close enough. smile.gif

Since this thread is huge and things can easily get lost in it I'll replicate my test results here (once again):

QUOTE (Alex B @ Aug 20 2008, 17:44) *
I finally had time to continue my bitrate tests with classsical music as I promised earlier in this thread (a thing called summer got in the way...)

After browsing through my lossless classical library I picked 25 "reference" tracks that should be quite representative. I avoided the extremely low and high lossless bitrates and tried to select tracks that have quite varied qualities.

Apparently iTunes has changed radically since my last test. Back then the 128 kbps VBR setting was suitable, but the 7.7 version uses bitrates in a more relaxed way and the 128 kbps VBR setting produces higher bitrates than before. Fortunately the 112 kbps VBR setting appears to be suitable for our test.
EDIT: As explained earlier in this thread, the fundamental difference in the bitrate behavior was found out to be caused by a bug in iTunes. However, the "iTunes 7.7" bitrates in my test are now assumed to be correct. Since my test Apple has released iTunes 8.0, but apparently its MP3 encoder has not changed. iTunes 7.7 and 8.0 create identical MP3 files.

In addition, I retested the "various" bitrates with the latest encoder versions when applicable.

Since I didn't have the old iTunes version installed I couldn't test the "classical" bitrates with it (which would have been unnecessary anyway).

FhG, iTunes and LAME 3.97 have only one suitable VBR setting for this test so I'd suggest to calculate the average bitrate of these encoders and adjust the Helix and LAME 3.98 settings to match this average. Helix -V60 and LAME -V5.7 appear to be pretty close to this average with my test tracks.

Here are the new results:

Summary




Various - table and chart






Classical - table and chart






EDIT

I forgot to mention that if anyone wants to test FhG's bitrate behavior the bitrates must be measured correctly. Most programs don't show accurate bitrate values because FhG doesn't write Xing headers to VBR files.

I used EncSpot Pro's "full scan" option.


This post has been edited by Alex B: Oct 16 2008, 13:18


--------------------
http://listening-tests.freetzi.com
Go to the top of the page
+Quote Post
Sebastian Mares
post Oct 16 2008, 13:24
Post #358





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Alex, I think this is the 4th time you quoted yourself. laugh.gif


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
Alex B
post Oct 16 2008, 18:01
Post #359





Group: Members
Posts: 1303
Joined: 14-September 05
From: Helsinki, Finland
Member No.: 24472



Is it? And they still don't learn...

Maybe I should put the test results presentation in my signature. tongue.gif


(Don't take this personally, Polar. It's just that the "test samples vs. complete library" bitrate difference issue has been discussed over and over again in the listening test threads.)


--------------------
http://listening-tests.freetzi.com
Go to the top of the page
+Quote Post
Sebastian Mares
post Oct 17 2008, 00:12
Post #360





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



http://www.listening-tests.info/mp3-128-1/

Please report any major bugs. I will make the public announcement tomorrow (actually today since it's after midnight already tongue.gif).

BTW, many thanks to rjamorim and Polar for hosting the samples!

This post has been edited by Sebastian Mares: Oct 17 2008, 00:15


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
hellokeith
post Oct 17 2008, 01:56
Post #361





Group: Members
Posts: 288
Joined: 14-August 06
Member No.: 34027



Just a grammar/punctuation nitpick:

The purpose of this test is to find out which popular MP3 VBR encoder outputs the best quality on bitrates around 128 kbps.

(removed comma and added MP3 smile.gif)
Go to the top of the page
+Quote Post
caligae
post Oct 17 2008, 08:22
Post #362





Group: Members
Posts: 186
Joined: 23-January 02
Member No.: 1132



From the readme:
QUOTE
Linux users are asked to use Wine with "wine wcmd /c DecodeXX.bat" from the "bin" directory.


Apparently "wcmd" was renamed to "cmd" in wine 0.9.21 which was released 2 years ago.

Is there any description of the samples available? E.g. if I want to test a specific genre I'd have to download all samples. Also, if one likes a track, it might be interesting what's the artist/title.
Go to the top of the page
+Quote Post
Sebastian Mares
post Oct 17 2008, 08:38
Post #363





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Thanks for the wmcd feedback. I will edit the readmes once I get home this evening.

As for the sample names - they will be published only after the test. With past tests I had the problem, that a lot of people focused on several samples for which I then had lots of results while other samples remained untested.

If a lot of people complain about this, I might publish them, though.

This post has been edited by Sebastian Mares: Oct 17 2008, 08:39


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post

15 Pages V  « < 13 14 15
Closed TopicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 29th November 2014 - 11:25