IPB

Welcome Guest ( Log In | Register )

MP3 encoders (HARD ROCK-ACOUSTIC) ABX listening tests (11 Jan 2012), Moderation: inconclusive ABX results (p-value > 0.05), no ABC/HR
zmejce
post Jan 11 2012, 23:59
Post #1





Group: Members
Posts: 17
Joined: 17-May 11
From: Rome
Member No.: 90728



Hello,
Ive tested mp3 encoders today and I like to share my results.
I havent done spectrum analisys, only ABX listening tests.

Programs used for encoding:
1. dbpoweramp 14.2 (with fraunhofer 4.0.3-latest)
2. iTunes 10.5.2.11 (uses fraunhofer mp3)
3. Easy CD-DA 15 (uses some version of lame 3.98)
4. foobar 1.1.8 (with lame3.99-20111018)
5. Nero 11 (uses fraunhofer mp3 encoder)

*note: even if there are few fraunhofer encoders, they are not exactly the same, also I've tested only 2 tracks of hard/glam rock genre (band name: Extreme). Ive chose that music because it has nice rhythm, drums and guitars, has nice effects on vocals and also contains samples of acoustic guitars and harmonics-and i didnt chose those tracks because of their names neither because of the lyrics that they are containing. Also i wanted to test WMP12's mp3 encoder, but i didnt found a way to encode a flac file to mp3 with it, also Ive found out that the flac decoder filter for wmp is with lower quality than other players (ex. winamp, foobar).

The settings used:
CBR 320kbps Joint Stereo, on itunes 10 ive checked "filter frequencies below 10Hz" & "smart encoding adjustments".


Filesize results:

_________Extreme1_______Extreme2
Flac_______26.2MB________27.4MB
Lame3.99__8.68MB________9.34MB
CD-DA_____8.69MB________9.35MB
Nero11_____8.68MB________9.34MB
iTunes10___8.68MB________9.34MB
dbPower___8.67MB________9.33MB

*note: iTunes tends to add 1-2 seconds of silence at the beggining and/or end of the track
Winner: dbPoweramp with fraunhofer

Encoding speed results:
Speed results were minimal in difference, the fastest was itunes, dbpoweramp was same or almost same as itunes, while the other encoders were few seconds late.
Winners: iTunes & dbPoweramp (both use fraunhofer)


ABX Listening Results:
1. Easy CD-DA
Extreme1
CODE
foo_abx 1.3.4 report
foobar2000 v1.1.8
2012/01/11 17:10:28

File A: D:\Hard rock\Extreme\1989 - Extreme\- 01 Little Girls.flac
File B: D:\Hard rock\Extreme\1989 - Extreme\cd-da\- 01 Little Girls.mp3

17:10:28 : Test started.
17:11:29 : 01/01 50.0%
17:12:12 : 02/02 25.0%
17:12:53 : 02/03 50.0%
17:13:26 : 02/04 68.8%
17:14:00 : 02/05 81.3%
17:14:05 : Test finished.

----------
Total: 2/5 (81.3%)


Extreme2
CODE
foo_abx 1.3.4 report
foobar2000 v1.1.8
2012/01/11 17:15:00

File A: D:\Hard rock\Extreme\1989 - Extreme\- 03 Kid Ego.flac
File B: D:\Hard rock\Extreme\1989 - Extreme\cd-da\- 03 Kid Ego.mp3

17:15:00 : Test started.
17:16:37 : 01/01 50.0%
17:16:49 : 02/02 25.0%
17:17:00 : 02/03 50.0%
17:17:15 : 03/04 31.3%
17:17:34 : 03/05 50.0%
17:18:42 : 03/06 65.6%
17:18:46 : Test finished.

----------
Total: 3/6 (65.6%)



Description:
This is my first ever ABX test I think of it as a warm-up smile.gif, all i remember that the difference was noticable and probably it was similar as lame3.99.


2. lame3.99-20111018
Extreme1
CODE
foo_abx 1.3.4 report
foobar2000 v1.1.8
2012/01/11 17:26:48

File A: D:\Hard rock\Extreme\1989 - Extreme\- 01 Little Girls.flac
File B: D:\Hard rock\Extreme\1989 - Extreme\lame3.99-20111018\- 01 Little Girls.mp3

17:26:48 : Test started.
17:27:16 : 01/01 50.0%
17:27:36 : 02/02 25.0%
17:27:54 : 03/03 12.5%
17:28:12 : 04/04 6.3%
17:28:31 : 04/05 18.8%
17:28:33 : Test finished.

----------
Total: 4/5 (18.8%)


Extreme2
CODE
foo_abx 1.3.4 report
foobar2000 v1.1.8
2012/01/11 17:29:28

File A: D:\Hard rock\Extreme\1989 - Extreme\- 03 Kid Ego.flac
File B: D:\Hard rock\Extreme\1989 - Extreme\lame3.99-20111018\- 03 Kid Ego.mp3

17:29:28 : Test started.
17:29:51 : 01/01 50.0%
17:30:09 : 02/02 25.0%
17:30:23 : 03/03 12.5%
17:30:36 : 04/04 6.3%
17:30:49 : 05/05 3.1%
17:30:50 : Test finished.

----------
Total: 5/5 (3.1%)



Description:
The Difference was very notable, especially on the first acoustic part (its in the first 15-20 secs of the song) of Extreme1 track and also on the on the vocals that had effects on the Extreme2 track. Absolutely the worst choice for compressing music to mp3.

3. Nero 11 (fraunhofer):
Extreme1
CODE
foo_abx 1.3.4 report
foobar2000 v1.1.8
2012/01/11 17:19:26

File A: D:\Hard rock\Extreme\1989 - Extreme\- 01 Little Girls.flac
File B: D:\Hard rock\Extreme\1989 - Extreme\nero\- 01 Little Girls.mp3

17:19:26 : Test started.
17:20:23 : 01/01 50.0%
17:20:57 : 01/02 75.0%
17:21:31 : 01/03 87.5%
17:22:57 : 02/04 68.8%
17:23:31 : 02/05 81.3%
17:23:33 : Test finished.

----------
Total: 2/5 (81.3%)


Extreme2
CODE
foo_abx 1.3.4 report
foobar2000 v1.1.8
2012/01/11 17:24:15

File A: D:\Hard rock\Extreme\1989 - Extreme\- 03 Kid Ego.flac
File B: D:\Hard rock\Extreme\1989 - Extreme\nero\- 03 Kid Ego.mp3

17:24:15 : Test started.
17:24:37 : 01/01 50.0%
17:24:57 : 01/02 75.0%
17:25:18 : 02/03 50.0%
17:25:36 : 03/04 31.3%
17:25:56 : 04/05 18.8%
17:26:14 : 04/06 34.4%
17:26:17 : Test finished.

----------
Total: 4/6 (34.4%)



Description:
Nero 11's fraunhofer encoder is a better choice than the lame versions used for testing. The result was very little noticable difference between the Flac's and the mp3's encoded with Nero11 .

4. iTunes 10.5.2 (fraunhofer)
Extreme1
CODE
foo_abx 1.3.4 report
foobar2000 v1.1.8
2012/01/11 17:31:31

File A: D:\Hard rock\Extreme\1989 - Extreme\- 01 Little Girls.flac
File B: D:\Hard rock\Extreme\1989 - Extreme\itunes\- 01 Little Girls.mp3

17:31:31 : Test started.
17:31:54 : 00/01 100.0%
17:32:13 : 01/02 75.0%
17:32:29 : 02/03 50.0%
17:32:47 : 02/04 68.8%
17:33:01 : 03/05 50.0%
17:33:03 : Test finished.

----------
Total: 3/5 (50.0%)


Extreme2
CODE
foo_abx 1.3.4 report
foobar2000 v1.1.8
2012/01/11 17:33:34

File A: D:\Hard rock\Extreme\1989 - Extreme\- 03 Kid Ego.flac
File B: D:\Hard rock\Extreme\1989 - Extreme\itunes\- 03 Kid Ego.mp3

17:33:34 : Test started.
17:34:04 : 00/01 100.0%
17:34:16 : 01/02 75.0%
17:34:36 : 02/03 50.0%
17:34:49 : 03/04 31.3%
17:35:11 : 04/05 18.8%
17:35:14 : Test finished.

----------
Total: 4/5 (18.8%)



Description:
iTunes mp3 encoder is very nice choice for encoding music. Althou its not absolutely best its sertanly better than lame. In the testings I was also able to hear minimal difference between the flac's and the mp3's encoded with iTunes.

5. dbPowerAmp (with fraunhofer)
Extreme1
CODE
foo_abx 1.3.4 report
foobar2000 v1.1.8
2012/01/11 17:35:45

File A: D:\Hard rock\Extreme\1989 - Extreme\- 01 Little Girls.flac
File B: D:\Hard rock\Extreme\1989 - Extreme\dbpoweramp\- 01 Little Girls.mp3

17:35:45 : Test started.
17:36:12 : 00/01 100.0%
17:36:34 : 00/02 100.0%
17:36:56 : 00/03 100.0%
17:37:23 : 00/04 100.0%
17:37:44 : 00/05 100.0%
17:37:46 : Test finished.

----------
Total: 0/5 (100.0%)


Extreme2
CODE
foo_abx 1.3.4 report
foobar2000 v1.1.8
2012/01/11 17:38:19

File A: D:\Hard rock\Extreme\1989 - Extreme\- 03 Kid Ego.flac
File B: D:\Hard rock\Extreme\1989 - Extreme\dbpoweramp\- 03 Kid Ego.mp3

17:38:19 : Test started.
17:38:36 : 00/01 100.0%
17:38:50 : 01/02 75.0%
17:39:04 : 01/03 87.5%
17:39:18 : 02/04 68.8%
17:39:35 : 03/05 50.0%
17:39:36 : Test finished.

----------
Total: 3/5 (50.0%)



Description:
dbPowerAmp's fraunhofer encoder is the absolute winner. I wasnt able to hear any difference between the flac's and the mp3's encoded with dbPowerAmp's fraunhofer encoder. Its definitely "ears bit perfect" encoder.

*note about ABX testings: on the first track (little girls) i was able to hear the difference from 4 of the encoders from the acoustic guitar in the beginning of the song-somewhere in the first 15-20 seconds. On the 2nd track (kid ego) i was able to hear the difference from 4 of the encoders at the beginning of the song when appeared vocals with effects. During the main part of the song and electric guitars, all encoders sounded identical or almost identical.

For the test ive used:
Core 2 Quad 2.5ghz, audigy platinum ex, audio system: panasonic sa-ak27 2x100W with superwoofer (its 8yrs old but it has quite good quality for home usage)
Ive used Foobar 1.1.8 with ABX component.


Conclusion:

dbPowerAmp with fraunhofer makes the mp3s smallest in size and best in quality-I couldnt hear any difference between its mp3s and the flacs-also its fastest or next to itunes, itunes is also excelent althou not "ears bit perfect" as dbPowerAmp, Nero is fine too, while lame is something that even at Highest quality settings provides lower quality than the other mp3 encoders.

Thanks for reading.
Go to the top of the page
+Quote Post
 
Start new topic
Replies
kornchild2002
post Jan 12 2012, 00:15
Post #2





Group: Members
Posts: 2080
Joined: 8-April 05
From: Cincinnati, OH
Member No.: 21277



I hate to say it but your testing was flawed. 5-6 trials for a single song isn't nearly enough. You should conduct the test and make at least 10 determinations as to which sone is which. 5 guesses is way too short as the results can be skewed. Increasing the sample number (i.e. how many times you pick which song is which) is necessary. People gave you links in your original post where you did absolutely no testing but it looks like you didn't fully read them.
Go to the top of the page
+Quote Post
zmejce
post Jan 12 2012, 00:18
Post #3





Group: Members
Posts: 17
Joined: 17-May 11
From: Rome
Member No.: 90728



QUOTE (kornchild2002 @ Jan 12 2012, 00:15) *
I hate to say it but your testing was flawed. 5-6 trials for a single song isn't nearly enough. You should conduct the test and make at least 10 determinations as to which sone is which. 5 guesses is way too short as the results can be skewed. Increasing the sample number (i.e. how many times you pick which song is which) is necessary. People gave you links in your original post where you did absolutely no testing but it looks like you didn't fully read them.



If i do 10-15 checks on every mp3 itll take me a week of testing. Btw it makes me tired and i think the results will be more inaccurate. 5-6 checks is pretty fine.
Go to the top of the page
+Quote Post
saratoga
post Jan 12 2012, 00:27
Post #4





Group: Members
Posts: 5047
Joined: 2-September 02
Member No.: 3264



QUOTE (zmejce @ Jan 11 2012, 18:18) *
5-6 checks is pretty fine.


CODE
17:35:45 : Test started.
17:36:12 : 00/01  100.0%
17:36:34 : 00/02  100.0%
17:36:56 : 00/03  100.0%
17:37:23 : 00/04  100.0%
17:37:44 : 00/05  100.0%
17:37:46 : Test finished.


You went 5 for 5 the wrong way . . . clearly if this happened you're not doing enough tests!

Go to the top of the page
+Quote Post
zmejce
post Jan 12 2012, 00:29
Post #5





Group: Members
Posts: 17
Joined: 17-May 11
From: Rome
Member No.: 90728



QUOTE (saratoga @ Jan 12 2012, 00:27) *
QUOTE (zmejce @ Jan 11 2012, 18:18) *
5-6 checks is pretty fine.


CODE
17:35:45 : Test started.
17:36:12 : 00/01  100.0%
17:36:34 : 00/02  100.0%
17:36:56 : 00/03  100.0%
17:37:23 : 00/04  100.0%
17:37:44 : 00/05  100.0%
17:37:46 : Test finished.


You went 5 for 5 the wrong way . . . clearly if this happened you're not doing enough tests!



That happened because theres no difference between dbPowerAmp's mp3 and the flac lol
Go to the top of the page
+Quote Post
saratoga
post Jan 12 2012, 00:34
Post #6





Group: Members
Posts: 5047
Joined: 2-September 02
Member No.: 3264



QUOTE (zmejce @ Jan 11 2012, 18:29) *
QUOTE (saratoga @ Jan 12 2012, 00:27) *
QUOTE (zmejce @ Jan 11 2012, 18:18) *
5-6 checks is pretty fine.

You went 5 for 5 the wrong way . . . clearly if this happened you're not doing enough tests!



That happened because theres no difference between dbPowerAmp's mp3 and the flac lol


If there is no difference, you should get 50/50, provided you do enough trails. You got 0/100 -> not enough trials.

Logically, if you can get 0/5 by chance, you can just as easily get 5/5 by chance. You need to do a lot more trials.




Go to the top of the page
+Quote Post
greynol
post Jan 12 2012, 00:37
Post #7





Group: Super Moderator
Posts: 10051
Joined: 1-April 04
From: San Francisco
Member No.: 13167



QUOTE (saratoga @ Jan 11 2012, 15:34) *
If there is no difference, you should get 50/50, provided you do enough trails.

You mean to say 50/100.

This post has been edited by greynol: Jan 12 2012, 00:37


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
saratoga
post Jan 12 2012, 00:39
Post #8





Group: Members
Posts: 5047
Joined: 2-September 02
Member No.: 3264



QUOTE (greynol @ Jan 11 2012, 18:37) *
QUOTE (saratoga @ Jan 11 2012, 15:34) *
If there is no difference, you should get 50/50, provided you do enough trails.

You mean to say 50/100.


Opps, just realized I used A/B and A/(A+B) in alternative paragraphs ohmy.gif
Go to the top of the page
+Quote Post

Posts in this topic
- zmejce   MP3 encoders (HARD ROCK-ACOUSTIC) ABX listening tests (11 Jan 2012)   Jan 11 2012, 23:59
- - kornchild2002   I hate to say it but your testing was flawed. 5-6...   Jan 12 2012, 00:15
|- - zmejce   QUOTE (kornchild2002 @ Jan 12 2012, 00:15...   Jan 12 2012, 00:18
|- - saratoga   QUOTE (zmejce @ Jan 11 2012, 18:18) 5-6 c...   Jan 12 2012, 00:27
||- - zmejce   QUOTE (saratoga @ Jan 12 2012, 00:27) QUO...   Jan 12 2012, 00:29
||- - saratoga   QUOTE (zmejce @ Jan 11 2012, 18:29) QUOTE...   Jan 12 2012, 00:34
||- - greynol   QUOTE (saratoga @ Jan 11 2012, 15:34) If ...   Jan 12 2012, 00:37
||- - saratoga   QUOTE (greynol @ Jan 11 2012, 18:37) QUOT...   Jan 12 2012, 00:39
|- - dhromed   First, zmejce, congratulations on having the testi...   Jan 12 2012, 10:06
- - db1989   That, and most of those percentages are so high as...   Jan 12 2012, 00:19
|- - zmejce   Well if you hear any difference in the sound once ...   Jan 12 2012, 00:25
- - greynol   All of the results of all the posted could have ju...   Jan 12 2012, 00:28
- - greynol   QUOTE (zmejce @ Jan 11 2012, 15:29) That ...   Jan 12 2012, 00:34
- - db1989   QUOTE (zmejce @ Jan 11 2012, 23:25) Well ...   Jan 12 2012, 00:35
- - pdq   My favorite part was ranking the encoders to see w...   Jan 12 2012, 04:00
- - IgorC   zmejce If You want to see which codec was better t...   Jan 12 2012, 04:33
- - Frank Bicking   Thread split: Minimum number of required ABX trial...   Jan 12 2012, 08:29
- - hlloyge   Well, I really can't see why zmejce is proud o...   Jan 12 2012, 09:45


Closed TopicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 26th October 2014 - 02:37