IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
My LAME tests
tr1n1tr0n
post Feb 18 2003, 22:33
Post #1





Group: Members
Posts: 1
Joined: 18-February 03
Member No.: 5062



Hi forum people,

After following the various posts arguing for and against the transparency of LAME and it's various bitrates and encoding methods, I decided to put my ears to the test and try a blind test with a variety of modes.

The piece of music used was track 5 from "Faith - A Message From The Spirits" on the Soul Jazz label, which is an album of religious music from around the world. More specifically, the piece in question is a solo percussion piece on Djouba, which sound more or less like bongos (to me anyway!) so as you can imagine, the music is fast with great transient attack.

Music ripped with EAC Secure/C2, encoded with LAME 3.94 a11 17Feb build, decoded using the MAD 0.14.2b plugin, listened to through an Echo Gina 20 digitally connected to a Marantz PM-75 amp, monitored on Sennheiser HD580 headphones. ABC/HR was used to audition the samples.

Results:

All cbr files are stereo, q2. All preset files use their respective settings in LAME.
"CBR 192 EAC Filt" is 192Kbit Stereo Q2, with a lowpass filter (19383-19916Hz) that EAC includes, even if no command line option is set.

ABC/HR Version 0.9b, 30 August 2002
Testname:

1R = D:\cdrips\test\preset 128.wav
2L = D:\cdrips\test\cbr 256 no filt.wav
3L = D:\cdrips\test\preset standard.wav
4L = D:\cdrips\test\preset cbr 192.wav
5L = D:\cdrips\test\cbr 192 eac filt.wav
6L = D:\cdrips\test\cbr 192 no filt.wav

---------------------------------------
General Comments:

---------------------------------------
1R File: D:\cdrips\test\preset 128.wav
1R Rating: 2.0
1R Comment:
---------------------------------------
2L File: D:\cdrips\test\cbr 256 no filt.wav
2L Rating: 4.7
2L Comment:
---------------------------------------
3L File: D:\cdrips\test\preset standard.wav
3L Rating: 4.4
3L Comment:
---------------------------------------
4L File: D:\cdrips\test\preset cbr 192.wav
4L Rating: 3.9
4L Comment:
---------------------------------------
5L File: D:\cdrips\test\cbr 192 eac filt.wav
5L Rating: 4.3
5L Comment:
---------------------------------------
6L File: D:\cdrips\test\cbr 192 no filt.wav
6L Rating: 4.6
6L Comment:
---------------------------------------


So, summing up, this test indicates to me that MP3, even if encoded by the latest LAME engine, is definitely not transparent up to and including 256Kbit. Not once was an MP3 mistakenly rated above the reference CD-Audio sample. The source material is very simple (solo bongo) which one would assume would be an easy task for an encoder. The standard preset fares quite well, but at the cost of file size (an 8% premium over 192Kbit CBR). Filtering does not benefit the sound, as shown by the difference in rating between the 5th and 6th samples, which are otherwise the same. To further investigate the comparison between filtered and non-filtered mp3, I made up a second ABC/HR test, this time comparing samples 5 and 6, the non-filtered sample as the reference, and the filtered sample as the comparator, 8 times over.

ABC/HR Version 0.9b, 30 August 2002
Testname:

1L = D:\cdrips\test\cbr 192 eac filt.wav
2L = D:\cdrips\test\cbr 192 eac filt.wav
3L = D:\cdrips\test\cbr 192 eac filt.wav
4R = D:\cdrips\test\cbr 192 eac filt.wav
5R = D:\cdrips\test\cbr 192 eac filt.wav
6L = D:\cdrips\test\cbr 192 eac filt.wav
7R = D:\cdrips\test\cbr 192 eac filt.wav
8L = D:\cdrips\test\cbr 192 eac filt.wav

---------------------------------------
General Comments:

---------------------------------------
1L File: D:\cdrips\test\cbr 192 eac filt.wav
1L Rating: 4.0
1L Comment:
---------------------------------------
2L File: D:\cdrips\test\cbr 192 eac filt.wav
2L Rating: 4.0
2L Comment:
---------------------------------------
3L File: D:\cdrips\test\cbr 192 eac filt.wav
3L Rating: 4.0
3L Comment:
---------------------------------------
4R File: D:\cdrips\test\cbr 192 eac filt.wav
4R Rating: 4.0
4R Comment:
---------------------------------------
5L File: D:\cdrips\test\cbr 192 no filt.wav
5L Rating: 4.0
5L Comment:
---------------------------------------
6L File: D:\cdrips\test\cbr 192 eac filt.wav
6L Rating: 4.0
6L Comment:
---------------------------------------
7R File: D:\cdrips\test\cbr 192 eac filt.wav
7R Rating: 4.0
7R Comment:
---------------------------------------
8L File: D:\cdrips\test\cbr 192 eac filt.wav
8L Rating: 4.0
8L Comment:
---------------------------------------

Only once did the filtered sample fare better than the untouched, unfiltered sound. So there you have it.
Go to the top of the page
+Quote Post
Secret Chief
post Feb 19 2003, 00:39
Post #2





Group: Members
Posts: 46
Joined: 23-January 03
Member No.: 4712



Yeah, MP3 certainly has its faults, which is why it's not recommended unless you need hardware compatability.

I did notice, however, that you used an alpha of LAME; the recommended version here is 3.90.2. You also didn't try alt-preset extreme, though that may not make much of a difference. Could you try it again with 3.90.2 or 3.92?

A solo bongo is actually quite hard for any encoder; the sound has quick transients and, worse still, has no masking noises around it.

Try your tests again with MPC, and see if it is transparent for you.
Go to the top of the page
+Quote Post
KikeG
post Feb 19 2003, 09:16
Post #3


WinABX developer


Group: Developer
Posts: 1578
Joined: 1-October 01
Member No.: 137



For the sake of rigorousness (statistical validity), I advise to try to ABX at least the highest quality encodes.
Go to the top of the page
+Quote Post
Garf
post Feb 19 2003, 09:21
Post #4


Server Admin


Group: Admin
Posts: 4885
Joined: 24-September 01
Member No.: 13



QUOTE (KikeG @ Feb 19 2003 - 10:16 AM)
For the sake of rigorousness (statistical validity), I advise to try to ABX at least the highest quality encodes.


He implicitly ABX'ed it 6/6 globally. But I suspect you mean doing an ABX of the highest scoring sample (CBR 256) seperately?
Go to the top of the page
+Quote Post
Garf
post Feb 19 2003, 09:27
Post #5


Server Admin


Group: Admin
Posts: 4885
Joined: 24-September 01
Member No.: 13



QUOTE
so as you can imagine, the music is fast with great transient attack.


You contradict yourself with this...

QUOTE
The source material is very simple (solo bongo)


..and this. Sharp transients are a known worst case scenario for MP3.

QUOTE
So, summing up, this test indicates to me that MP3, even if encoded by the latest LAME engine, is definitely not transparent up to and including 256Kbit.


Not for all music out there in existence, no. No lossy codec can ever always be transparent, and I doubt you'll find anyone making that claim.

We are generally not talking about absolute transparency since it's an unattainable dream. But LAME APS for example will generate MP3's that are indistinguishable from the CD for almost anything out there. That doesn't mean there aren't exceptions.

I wonder what your point was? MP3 is not perfect? Well, no.
Go to the top of the page
+Quote Post
KikeG
post Feb 19 2003, 10:59
Post #6


WinABX developer


Group: Developer
Posts: 1578
Joined: 1-October 01
Member No.: 137



QUOTE (Garf @ Feb 19 2003 - 09:21 AM)
QUOTE (KikeG @ Feb 19 2003 - 10:16 AM)
For the sake of rigorousness (statistical validity), I advise to try to ABX at least the highest quality encodes.


He implicitly ABX'ed it 6/6 globally. But I suspect you mean doing an ABX of the highest scoring sample (CBR 256) seperately?

yes, I'd try to ABX with p<5% (better p<1%) for individual encodes vs. original, at least the better scored ones (CBR256, CBR192) and APS. Maybe the better encodes were correctly identified by chance. Also, in my experience CBR256 in Lame 3.90.2 is always worse than APS in pre-echo, but maybe this changes in Lame 3.94?

Another round of abc/hr to check consistency of results would be good too.
Go to the top of the page
+Quote Post
Garf
post Feb 19 2003, 11:30
Post #7


Server Admin


Group: Admin
Posts: 4885
Joined: 24-September 01
Member No.: 13



QUOTE (KikeG @ Feb 19 2003 - 11:59 AM)
Also, in my experience CBR256 in Lame 3.90.2 is always worse than APS in pre-echo, but maybe this changes in Lame 3.94?

I don't know - it would be good to use the latest trusted encoder anyway (3.90.2), you can get flawed results very easily with alpha versions.
Go to the top of the page
+Quote Post
LordofStars
post Feb 19 2003, 18:36
Post #8





Group: Members
Posts: 353
Joined: 28-April 02
Member No.: 1894



Some people have found the preecho of standard from 3.94 to be worse than 3.90.2


--------------------
r3mix zealot.
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 16th September 2014 - 13:02