IPB

Welcome Guest ( Log In | Register )

3 Pages V   1 2 3 >  
Reply to this topicStart new topic
LAME 3.96 vs. 3.90.3 Test, Test/Result Thread
dev0
post Mar 17 2004, 18:31
Post #1





Group: Developer
Posts: 1679
Joined: 23-December 01
From: Germany
Member No.: 731



The purpose of this thread is to (finally) test LAME 3.96 throughly enough to make it the new recommended LAME version and enhance communication between LAME developers and the HA.org community. The recent confusion about the different LAME versions and compiles (-Qrcd trouble etc.) have led to the decision that moving along with LAME development has been postponed for long enough and LAME 3.90.X can hopefully be declared dead very soon.
Please test as many samples as possible according to the following guidelines and post any results here.


1. Use the following LAME compiles (updated!):
lame3.90.3
lame3.96b2

2. The focus of the test should be --alt-preset/--preset standard, since it will allow us to make conclusions regarding the overall performance of the 'code level tweaked' VBR presets. Other VBR/ABR/CBR presets are interesting too, but not as important. If problems with --alt preset standard are detected feel free to compare extreme and insane too. Test the following combinations please:

(alt)presets + VBR/ABR
(320kbps) 3.96 --preset insane vs. 3.90.3 --alt-preset insane
(~256kbps) 3.96 --preset extreme vs. 3.90.3 --alt-preset extreme
(~210kbps) 3.96 --preset standard vs. 3.90.3 --alt-preset standard
(~160kbps) 3.96 -V 4 vs. 3.96 --preset 160 vs. 3.90.3 --alt-preset 160
(~128kbps) 3.96 -V 5 vs. 3.96 --preset 128 vs. 3.90.3 --alt-preset 128

CBR
3.96 --preset cbr <bitrate> vs. 3.90.3 --alt-preset cbr <bitrate>

If you want, you can additionally test VBR/ABR vs. CBR at comparable bitrates

3. You may use any sample you want, as long as you upload a losslessly compressed version of that (or provide a working link), so others can verify your results.

4. Your test results have to include the following:
  • ABX results for
    3.90.3 vs. Original
    3.96b2 vs. Original
    3.96b2 vs. 3.90.3
  • ABC/HR results are appreciated especially at lower bitrates, but shouldn't be considered a requirement.
  • (Short) descriptions of the artifacts/differences
Notes:
This thread is for results only, discuss about the test here, please.
The earlier 'regression examples" thread has been closed and we'll try to add all results from it to the collected test results below. The original thread is still available here.
Please provide direct links to the sample you tested with or upload it here, even if it's a well-known one - it'll make things much easier for others.


Edit by tigre, Mar 31 2004: Updated to 3.96b2 and modified/simplified medium bitrate ABR/VBR test recommendations according to 3.96 -V x mass encode average bitrates.


--------------------
"To understand me, you'll have to swallow a world." Or maybe your words.
Go to the top of the page
+Quote Post
dev0
post Mar 18 2004, 14:43
Post #2





Group: Developer
Posts: 1679
Joined: 23-December 01
From: Germany
Member No.: 731



Test results so far:


320kbps

3.96b1 --p insane < 3.90.3 --ap insane :: Lazy_Jones :: owowo :: 0x verified so far

~ 256kbps
3.96b1 --p extreme < 3.90.3 --ap extreme :: Lazy_Jones :: owowo :: 0x verified so far

~ 210kbps
3.96b1 --p standard < 3.90.3 --ap standard :: myf_4sec :: LoFiYo :: 2x verified by Wombat, harashin
3.96b2 : 1x verified by LoFiYo
3.96b1 --p standard < 3.90.3 --ap standard :: Rebel :: Proxima :: 1x verified by tigre
3.96b1 --p standard < 3.90.3 --ap standard :: (-) Ions :: Westgroveg :: 1x verified so far by harashin
3.96b1 --p standard < 3.90.3 --ap standard :: Hustlejet :: harashin :: 0x verified so far
3.96b2 : 1x verified by harashin
3.96b1 --p standard < 3.90.3 --ap standard :: trumpets1 :: SometimesWarrior :: 0x verified so far
3.96b1 --p standard < 3.90.3 --ap standard :: Doesnair :: Moitah :: 1x verified by harashin
3.96b1 --p standard < 3.90.3 --ap standard :: cantwait :: tigre :: 2x verified by harashin, Moitah
3.96b1 --p standard < 3.90.3 --ap standard :: death2 :: SometimesWarrior :: 1x verified by harashin
3.96b1 --p standard < 3.90.3 --ap standard :: Destitute :: harashin :: 0x verified so far
3.96b1 --p standard < 3.90.3 --ap standard :: Rosemary :: harashin :: 0x verified so far
3.96b1 --p standard < 3.90.3 --ap standard :: drone_short :: freakngoat :: 0x verified so far
3.96b1 --p standard < 3.90.3 --ap standard :: Chanchan1 :: tigre :: 1x verified by harashin
3.96b2 : 1x verified by tigre
3.96b1 --p standard < 3.90.3 --ap standard :: Hosokawa___Atem_lied :: harashin :: 0x verified so far
3.96b2 : 1x verified by harashin
3.96b2 --p standard = 3.90.3 --ap standard :: 41_30sec :: ViPER1313
3.96b1 --p standard > 3.90.3 --ap standard :: spahm :: Pio2001 :: 1x verified by Wombat
3.96b1 --p standard > 3.90.3 --ap standard :: Birds :: Wombat :: 0x verified so far
3.96b1 --p standard > 3.90.3 --ap standard :: hokuscaredpiano :: Moitah :: 0x verified so far
3.96b1 --p standard > 3.90.3 --ap standard :: Lazy_Jones :: owowo :: 1x verified by freakngoat

~ 192kbps
no results so far

~ 160kbps
3.96b2 --p 160 > 3.90.3 --ap 160 > 3.96b2 -V 4 :: myf_4sec :: LoFiYo :: 0x verified so far + missing ABX results*

~ 144kbps
no results so far

~ 128kbps
VBR/ABR

3.96b1 --p 128 < 3.90.3 --ap 128 :: Quizas :: tigre :: 0x verified so far
3.96b1 --p 128 < 3.90.3 --ap 128 :: Doesnair :: Moitah :: 0x verified so far + missing ABX results*
3.96b1 --p 128 < 3.90.3 --ap 128 :: (-) Ions :: PVNC :: 0x verified so far + missing ABX results*
3.96b1 --p 128 < 3.90.3 --ap 128 :: Applaud :: [proxima] :: 0x verified so far + missing ABX results*
3.96b1 --p 128 < 3.90.3 --ap 128 :: Bassdrum :: [proxima] :: 0x verified so far + missing ABX results*
3.96b1 --p 128 < 3.90.3 --ap 128 :: Blackwater :: [proxima] :: 0x verified so far + missing ABX results*
3.96b1 --p 128 < 3.90.3 --ap 128 :: Campestre :: [proxima] :: 0x verified so far + missing ABX results*
3.96b1 --p 128 < 3.90.3 --ap 128 :: Fall :: [proxima] :: 0x verified so far + missing ABX results*
3.96b1 --p 128 < 3.90.3 --ap 128 :: Iron :: [proxima] :: 0x verified so far + missing ABX results*
3.96b1 --p 128 < 3.90.3 --ap 128 :: Thewayitis :: [proxima] :: 0x verified so far + missing ABX results*
3.96b1 --p 128 < 3.90.3 --ap 128 :: Tosca :: [proxima] :: 0x verified so far + missing ABX results*
3.96b1 --p 128 > 3.90.3 --ap 128 :: fatboy :: sony666 :: 2x verified so far by Wombat, FatBoyFin
3.96b1 --p 128 -q 0 < 3.90.3 --ap 128 :: Velvet :: [proxima] :: 0x verified so far + missing ABX results*
3.96b1 -V 5 > 3.90.3 --ap 128 :: Quizas :: tigre :: 0x verified so far
3.96b2 --V 5 > 3.96b1 --p 128 > 3.90.3 --ap 128 :: fatboy :: FatBoyFin :: 0x verified so far
3.96b2 --V 5 > 3.90.3 --ap 128 > 3.96b1 --p 128 :: Its_me :: tigre :: 0x verified so far
3.96b2 --V 5 > 3.90.3 --ap 128 > 3.96b1 --p 128 :: entierren con rumba :: tigre :: 0x verified so far

CBR
3.96b1 --p cbr 128 < 3.90.3 --ap cbr 128 :: Quizas :: tigre :: 0x verified so far

* If a result doesn't match the minimum requirements (-> 4. in 1st post), the result is 'greyed out' until the missing data is provided or someone else confirms the results. If your result isn't included in this list at all, we're either too slow, or there's too much missing (ABX results, link to the sample, clear statment which version is better, information about lame version/setting used etc.). If you want to provide missing information, do it in a new post to ensure that we notice it.

This post has been edited by tigre: Apr 14 2004, 13:37


--------------------
"To understand me, you'll have to swallow a world." Or maybe your words.
Go to the top of the page
+Quote Post
sony666
post Mar 18 2004, 22:50
Post #3





Group: Members
Posts: 573
Joined: 22-February 02
Member No.: 1375



3.96b1 shows great improvement with fatboy and --alt-preset 128 (ABR)

-wav vs 3.90.3: 8/8, horribly distorted "vocals", "knocking" artifact 1.5-3.0s
-wav vs 3.96b1: 8/8 badly distorted vocals
-3.90.3 vs 3.96b1: 8/8, clearly less distortion in 3.96b1

I tried aps, but failed miserably on both encoders this time.
Go to the top of the page
+Quote Post
LoFiYo
post Mar 19 2004, 06:53
Post #4





Group: Members
Posts: 133
Joined: 2-January 04
Member No.: 10896



I used this to test APS 3.90.3 vs APS 3.96b1.

3.90.3 APS was 138kbps; 3.96b1 APS was 116kbps. If you don't have a golden set of ears, I recommend you try this sample. It's very easy to ABX tongue.gif .

ABC/HR Version 0.9b, 30 August 2002
Testname: My Funny Valentine - 3.90.3 vs 3.96b1

1R = C:\My Music\lab\MYF-4SEC\aps396b1.wav
2R = C:\My Music\lab\MYF-4SEC\aps3903.wav

---------------------------------------
General Comments:
#2 is the obvious winner.
---------------------------------------
1R File: C:\My Music\lab\MYF-4SEC\aps396b1.wav
1R Rating: 1.0
1R Comment: Distortion is very noticeable throughout the sample.
---------------------------------------
2R File: C:\My Music\lab\MYF-4SEC\aps3903.wav
2R Rating: 3.9
2R Comment: Much better than #1. Perceptible and almost not annoying.
---------------------------------------
ABX Results:
Original vs C:\My Music\lab\MYF-4SEC\aps396b1.wav
8 out of 8, pval = 0.004
Original vs C:\My Music\lab\MYF-4SEC\aps3903.wav
8 out of 8, pval = 0.004
C:\My Music\lab\MYF-4SEC\aps396b1.wav vs C:\My Music\lab\MYF-4SEC\aps3903.wav
8 out of 8, pval = 0.004
Go to the top of the page
+Quote Post
Wombat
post Mar 19 2004, 23:14
Post #5





Group: Members
Posts: 1034
Joined: 7-October 01
Member No.: 235



Just verified myf_4sec with aps

I could ABX 396b -> original 8/8
and 3903 -> original 5/8

As this sample is new to me i am surprised how obvious 396b suffers here with the third blow of the trumpet!

Wombat
Go to the top of the page
+Quote Post
Pio2001
post Mar 20 2004, 00:24
Post #6


Moderator


Group: Super Moderator
Posts: 3936
Joined: 29-September 01
Member No.: 73



I begin with the easiest samples for me. I included MPC out of curiosity, and WMA9 for reasons that are not necessary to explain here.

Lame 3.90.3 --alt-preset standard
Lame 3.96b1 --preset standard
Mppenc 1.14 --xlevel
WMA9 VBR 135-215 kbps

Sample : Badvilbel
Wav vs 3.90.3 ABX 11/11 Lots of pre echo, loss of transients, drop outs
Wav vs 3.96b1 ABX 8/8 Same as above
Wav vs MPC ABX 7/8 Gurgling on transients, ringing, drop outs, but better
Wav vs WMA9 ABX 8/8 Best, only transient problems, no drop outs at all

3.96.1 vs WMA9 : ABX 7/8
3.96.1 vs MPC : ABX 8/8

Conclusion of 3.90.3 vs 3.96b1 :
Same quality


Sample : Drone Short
3.90.3 : ABX 16/16 toooo easy ! Artifacts during the first half second, and time smearing during the second half.
3.96b1 : ABX 16/16 Somewhat louder artifacts, but not enough to talk about regression, just chance.
90 vs 96 : 11/14 (3% chance of guessing). The artifacts are not at the same place.
MPC : perfect. Maybe a tiny noise ? ABX 3/8, no.
WMA9 : 16/16, unacceptable quality

Conclusion of 3.90.3 vs 3.96b1 :
Same quality


Spahm
3.90.3 : Noise. 16/16
3.96.1 : perfect. Maybe a tiny noise ? 8/8 It seems so...
3.90.3 vs 3.96.1 : 16/16
MPC : Same effect as transients in badvilbel : grungy instead of hissy (mp3) 16/16
WMA : worse. 16/17 (I went too fast and made one mistake).

Conclusion of 3.90.3 vs 3.96b1 :
Big improvement Sample nearly completely solved.


For these three samples, compared to 3.90.3 :
3.96b1 is better one time, equal two times
MPC equal one time, slightly better one time, better one time.
WMA9 is unlistenable one time, worse one time, better one time.

Overall concusion : none for the time being. Results are sample dependant.
More tests to come, but with more difficult samples. It will take longer.

These samples are available on FF123's page : http://www.ff123.net/samples.html
Go to the top of the page
+Quote Post
Wombat
post Mar 20 2004, 00:44
Post #7





Group: Members
Posts: 1034
Joined: 7-October 01
Member No.: 235



I wrote befor in another thread about the birds sample and aps

3903 -> original 16/16 clear artifact when she sings the first "e" of become
396b only guessing
396b -> 3.903 8/8

396b is the first aps without a problem to me and birds.

The sample itself is now located here:
http://www.hydrogenaudio.org/forums/index....=0&#entry195385

I did 16 this time, for trying how long i can clearly discern. I am not used to use abx tests much.
I have to admit it is not that easy, even when you know the problem in the test file.

Edit: location of the sample file

Wombat

This post has been edited by Wombat: Mar 20 2004, 18:18
Go to the top of the page
+Quote Post
[proxima]
post Mar 20 2004, 17:26
Post #8





Group: Members
Posts: 197
Joined: 12-October 02
From: Italy
Member No.: 3537



Sample: rebel.wav
Artifact: Preecho.
ABX results (range 0:01-0:03 sec.):

Original vs. 3.90.2 (--aps -Z) 13/14
Original vs. 3.96b1 (--aps) 10/10
3.90.2 vs 3.96b1 10/10

Both non-transparent but 3.90.2 is slightly better.
As already reported in my previous test, the same sample also reveal ringing problem with 3.96b1 (--ap 128).
The sample is available in the apposite thread.

This post has been edited by [proxima]: Mar 20 2004, 17:28


--------------------
WavPack 4.3 -mfx5
LAME 3.97 -V5 --vbr-new --athaa-sensitivity 1
Go to the top of the page
+Quote Post
tigre
post Mar 20 2004, 21:56
Post #9


Moderator


Group: Members
Posts: 1434
Joined: 26-November 02
Member No.: 3890



Sample: Quizas
Tested lame settings:
3.90.3 --alt-preset 128 (132kbps)
3.90.3 --alt-preset cbr 128 (128kbps)
3.96b1 --preset 128 (135kbps)
3.96b1 --preset cbr 128 (128kbps)
3.96b1 -V 5 (138kbps)
Decoded with fb2k with trackgain enabled
Ratings:
3.96b1 -V 5 >> 3.90.3 --alt-preset 128 > 3.90.3 --alt-preset cbr 128 >> 3.96b1 --preset 128 > 3.96b1 --preset cbr 128

ABC/HR results:
QUOTE
ABC/HR Version 0.9b, 30 August 2002
Testname: Quizas 128kbps

1L = quizas.wav_3903_ap128.wav
2R = quizas.wav_396b1_pcbr128.wav
3R = quizas.wav_396b1_V5.wav
4L = quizas.wav_396b1_p128.wav
5L = quizas.wav_3903_apcbr128.wav

---------------------------------------
General Comments:
Ratings bases on 1st half (0.0-14.0), biggest problems there.
---------------------------------------
1L File: quizas.wav_3903_ap128.wav
1L Rating: 3.0
1L Comment: pre-echo/smearing, easy to ABX, e.g. 8.6-10.9
---------------------------------------
2R File: quizas.wav_396b1_pcbr128.wav
2R Rating: 1.0
2R Comment: pre-echo + ringing/chirping, easy to ABX, e.g. 8.6-10.9
---------------------------------------
3R File: quizas.wav_396b1_V5.wav
3R Rating: 4.0
3R Comment: only very small smearing of percussions, ABXed 8.6-10.9
---------------------------------------
4L File: quizas.wav_396b1_p128.wav
4L Rating: 1.5
4L Comment: preecho and hf chirping/warbling, ABXed 5.1-8.6
---------------------------------------
5L File: quizas.wav_3903_apcbr128.wav
5L Rating: 2.5
5L Comment: preecho, easy to ABX, e.g. 8.6-10.9
---------------------------------------
ABX Results:
Original vs quizas.wav_3903_ap128.wav
    10 out of 11, pval = 0.006
Original vs quizas.wav_396b1_pcbr128.wav
    7 out of 7, pval = 0.008
Original vs quizas.wav_396b1_V5.wav
    11 out of 12, pval = 0.003
Original vs quizas.wav_396b1_p128.wav
    7 out of 7, pval = 0.008
Original vs quizas.wav_3903_apcbr128.wav
    7 out of 7, pval = 0.008
quizas.wav_3903_ap128.wav vs quizas.wav_396b1_pcbr128.wav
    11 out of 12, pval = 0.003
quizas.wav_3903_ap128.wav vs quizas.wav_396b1_V5.wav
    7 out of 7, pval = 0.008
quizas.wav_3903_ap128.wav vs quizas.wav_396b1_p128.wav
    7 out of 7, pval = 0.008
quizas.wav_3903_ap128.wav vs quizas.wav_3903_apcbr128.wav
    11 out of 12, pval = 0.003
quizas.wav_396b1_pcbr128.wav vs quizas.wav_396b1_V5.wav
    7 out of 7, pval = 0.008
quizas.wav_396b1_pcbr128.wav vs quizas.wav_396b1_p128.wav
    8 out of 8, pval = 0.004
quizas.wav_396b1_pcbr128.wav vs quizas.wav_3903_apcbr128.wav
    7 out of 7, pval = 0.008
quizas.wav_396b1_V5.wav vs quizas.wav_396b1_p128.wav
    7 out of 7, pval = 0.008
quizas.wav_396b1_V5.wav vs quizas.wav_3903_apcbr128.wav
    7 out of 7, pval = 0.008
quizas.wav_396b1_p128.wav vs quizas.wav_3903_apcbr128.wav
    11 out of 12, pval = 0.003


--------------------
Let's suppose that rain washes out a picnic. Who is feeling negative? The rain? Or YOU? What's causing the negative feeling? The rain or your reaction? - Anthony De Mello
Go to the top of the page
+Quote Post
Moitah
post Mar 20 2004, 22:16
Post #10





Group: Members
Posts: 193
Joined: 5-June 02
From: Virginia Beach, VA
Member No.: 2227



QUOTE (dev0 @ Mar 18 2004, 08:43 AM)
3.96 --p standard < 3.90.3 --ap standard :: isitloveintro :: Moitah :: 0x verified so far + missing ABX results*

Could you remove this one? It isn't worse in 3.96, I was able to ABX for 3.90.3 as well (see further down in the original thread).

QUOTE (dev0 @ Mar 18 2004, 08:43 AM)
3.96 --p standard < 3.90.3 --ap standard :: Doesnair :: Moitah :: 0x verified so far + missing ABX results*

This sample is almost certainly a regression... I will do more thorough testing to make sure. Can someone else verify this (listen where the snair is hit, and its a good idea to start with --p 128 because the artifact is a lot more obvious)?
Go to the top of the page
+Quote Post
Moitah
post Mar 20 2004, 23:04
Post #11





Group: Members
Posts: 193
Joined: 5-June 02
From: Virginia Beach, VA
Member No.: 2227



Sample: doesnair
3.90.3 --ap standard vs 3.96b1 --p standard

CODE
Original vs H:\lame\doesnair\1579_doesnair-390-s.wav
   14 out of 16, pval = 0.002
Original vs H:\lame\doesnair\1583_doesnair-396-s.wav
   16 out of 16, pval < 0.001
H:\lame\doesnair\1579_doesnair-390-s.wav vs H:\lame\doesnair\1583_doesnair-396-s.wav
   13 out of 16, pval = 0.011


Both aren't transparent vs original, but 3.96b1 is worse.
Go to the top of the page
+Quote Post
Wombat
post Mar 20 2004, 23:33
Post #12





Group: Members
Posts: 1034
Joined: 7-October 01
Member No.: 235



spahm
This sample makes my ears vibrating somehow after too many tries!

3903 8/8 pretty easy i only listen the first beats and the added noise is pretty obvious
396b 7/8 the same noise but to a much lesser degree


hustlejet

just gave me helpless clicky, clicky and when i thought i heard the problem - Nada!
Not my cup of sample sad.gif

btw. ff123 did a nice job with abchr!!

Wombat
Go to the top of the page
+Quote Post
Moitah
post Mar 20 2004, 23:41
Post #13





Group: Members
Posts: 193
Joined: 5-June 02
From: Virginia Beach, VA
Member No.: 2227



Sample: hokuscaredpiano (Hoku - I'm Scared)
3.96b1 --p standard vs 3.90.3 --ap standard

CODE
Original vs H:\lame\hokuscaredpiano\1586_hokuscaredpiano-396-s.wav
   15 out of 16, pval < 0.001
Original vs H:\lame\hokuscaredpiano\1585_hokuscaredpiano-390-s.wav
   14 out of 16, pval = 0.002
H:\lame\hokuscaredpiano\1586_hokuscaredpiano-396-s.wav vs H:\lame\hokuscaredpiano\1585_hokuscaredpiano-390-s.wav
   13 out of 16, pval = 0.011


I'm hearing a problem on the piano (first chord change, 1.87 secs in), 3.96b1 sounds a bit better smile.gif.
Go to the top of the page
+Quote Post
tigre
post Mar 20 2004, 23:47
Post #14


Moderator


Group: Members
Posts: 1434
Joined: 26-November 02
Member No.: 3890



cantwait

I wanted to use this CD for testing lower bitrates but at 128kbps the artifacts were so obvious that I decided to start with (alt)preset standard.

Both, 3.90.3 --alt-preset standard and 3.96b1 --preset standard make the singer's voice (or it's echo) sound metallic in many places. One easily noticable example is the "s" of the word "understand" at ~ 8 seconds. Other "s" sounds (but also other places) have similar problems. 3.96b1 is clearly worse.

ABXed with fb2k, original vs. mp3 9/9 each time, 3.90.3 vs. 3.96b1 11/12

EDIT:
Edited in Wombat's + Moitah's (and my) results.

P.S.: Can someone verify my findings, please - is it just me or can this be considered as lame problem/killer sample?

This post has been edited by tigre: Mar 21 2004, 00:10


--------------------
Let's suppose that rain washes out a picnic. Who is feeling negative? The rain? Or YOU? What's causing the negative feeling? The rain or your reaction? - Anthony De Mello
Go to the top of the page
+Quote Post
harashin
post Mar 21 2004, 02:30
Post #15





Group: Members
Posts: 339
Joined: 20-February 02
From: Kyoto, Japan
Member No.: 1362



QUOTE (Wombat @ Mar 21 2004, 07:33 AM)
hustlejet

just gave me helpless clicky, clicky and when i thought i heard the problem - Nada!
Not my cup of sample sad.gif

That's bad ohmy.gif, it seems even ReplayGain didn't fix its clipping issue(I don't notice here though) on your system.

Anyway, I was able to confirm tigre's cantwait sample. Somehow I can't reach my hosting space at CyberQuébec today, I write these results here.
CODE
A file: I:\test\cantwait\cantwait.wav B file: I:\test\cantwait\3.96b1aps.wav
10/10  p< 0.1%
A file: I:\test\cantwait\cantwait.wav B file: I:\test\cantwait\3.90.3aps.wav
4/15  p=98.2%
A file: I:\test\cantwait\3.90.3aps.wav B file: I:\test\cantwait\3.96b1aps.wav
15/20  p=2.1%

EDIT: The server is back now. Results

This post has been edited by harashin: Mar 21 2004, 03:35


--------------------
Folding@Home Hydrogenaudio.org Team ID# 32639
http://folding.stanford.edu/
Go to the top of the page
+Quote Post
Moitah
post Mar 21 2004, 03:59
Post #16





Group: Members
Posts: 193
Joined: 5-June 02
From: Virginia Beach, VA
Member No.: 2227



QUOTE (tigre @ Mar 20 2004, 05:47 PM)
P.S.: Can someone verify my findings, please - is it just me or can this be considered as lame problem/killer sample?

Yeah, both are pretty easy to ABX against the original (--(alt-)preset standard):

CODE
Original vs H:\lame\1583_Max - Cant Wait Until Tonight (Dry Wurlitzer Mix)-396-s.wav
   18 out of 20, pval < 0.001
Original vs H:\lame\1585_Max - Cant Wait Until Tonight (Dry Wurlitzer Mix)-390-s.wav
   17 out of 20, pval = 0.001

Didn't have any luck ABXing them against eachother, but I wasn't concentrating as hard as I normally do... I'll try again later.

EDIT: Also, I only tried one part of the sample (the "s" in "understand").

EDIT 2: Took a break and switched to Grado SR225s (was using PortaPros before):

CODE
H:\lame\1585_Max - Cant Wait Until Tonight (Dry Wurlitzer Mix)-390-s.wav vs H:\lame\1583_Max - Cant Wait Until Tonight (Dry Wurlitzer Mix)-396-s.wav
   20 out of 25, pval = 0.002

3.96b1 is worse.

This post has been edited by Moitah: Mar 21 2004, 04:43
Go to the top of the page
+Quote Post
owowo
post Mar 21 2004, 06:48
Post #17





Group: Members
Posts: 7
Joined: 6-March 04
Member No.: 12526



Sample used: Lazy_Jones.flac
3.96b1 --preset standard <268kbps> vs. 3.90.3 --alt-preset standard <242kbps>
EDIT: Average bitrates according to Nero6
Encoded with lame.exe/LAMEDrop1.3 - Decoded with Nero6
CODE
-------------------------------------
WinABX v0.42 test report
03/21/2004 00:10:57

A file: C:\WINDOWS\Desktop\Lazy_Jones (wav).wav
B file: C:\WINDOWS\Desktop\Lazy_Jones 3.90.3 (--ap s).wav

00:11:59    1/1  p=50.0%
00:12:10    2/2  p=25.0%
00:12:19    3/3  p=12.5%
00:12:25    4/4  p=6.2%
00:12:29    5/5  p=3.1%
00:12:36    6/6  p=1.6%
00:12:40    7/7  p=0.8%
00:12:44    8/8  p=0.4%
00:12:55    9/9  p=0.2%
00:13:50  reset

00:14:18  test finished

-------------------------------------
WinABX v0.42 test report
03/21/2004 00:14:18

A file: C:\WINDOWS\Desktop\Lazy_Jones (wav).wav
B file: C:\WINDOWS\Desktop\Lazy_Jones 3.96b1 (--p s).wav

00:14:48    1/1  p=50.0%
00:14:58    1/2  p=75.0%
00:15:07    2/3  p=50.0%
00:15:16    3/4  p=31.2%
00:15:23    4/5  p=18.8%
00:15:31    5/6  p=10.9%
00:15:39    6/7  p=6.2%
00:15:49    7/8  p=3.5%
00:15:58    7/9  p=9.0%
00:16:28  reset

00:16:32  test finished

-------------------------------------
WinABX v0.42 test report
03/21/2004 00:17:47

A file: C:\WINDOWS\Desktop\Lazy_Jones 3.90.3 (--ap s).wav
B file: C:\WINDOWS\Desktop\Lazy_Jones 3.96b1 (--p s).wav

00:18:03    1/1  p=50.0%
00:18:12    2/2  p=25.0%
00:18:21    3/3  p=12.5%
00:18:30    4/4  p=6.2%
00:18:38    5/5  p=3.1%
00:18:47    6/6  p=1.6%
00:18:58    7/7  p=0.8%
00:19:09    8/8  p=0.4%
00:19:19    9/9  p=0.2%
00:19:22  reset

00:19:33  test finished

Comments:
3.90.3 --alt-preset standard; completely chokes on this one
3.96b1 --preset standard; near perfect accuracy

This post has been edited by owowo: Mar 21 2004, 11:25
Go to the top of the page
+Quote Post
[proxima]
post Mar 21 2004, 14:03
Post #18





Group: Members
Posts: 197
Joined: 12-October 02
From: Italy
Member No.: 3537



As requested by tigre i have to provide abx results for my listening test even if samples are very easy because there are few people that test a specific sample. I'm sorry if i'm too lazy to abx the two lame versions each other fot all the samples but you can always see ratings in the posted table and comments. For the great majority of the samples 3.90.2 is better. Most samples should be well known to testers, nevertheless i uploaded some of them.

[quote]Original vs applaud396q0.wav
    8 out of 8, pval = 0.004
Original vs applaud_3902.wav
    11 out of 12, pval = 0.003
Original vs applaud_396b1.wav
    10 out of 10, pval < 0.001[/quote]
available here: http://www.mp3dev.org/mp3/gpsycho/quality.html
[quote]Original vs Bassdrum_3902.wav
    10 out of 10, pval < 0.001
Original vs Bassdrum396q0.wav
    8 out of 8, pval = 0.004
Original vs Bassdrum_396b1.wav
    8 out of 8, pval = 0.004[/quote]
sample uploaded
[quote]Original vs Blackwater_396b1.wav
    8 out of 8, pval = 0.004
Original vs Blackwater_3902.wav
    8 out of 8, pval = 0.004
Original vs Blackwater396q0.wav
    8 out of 8, pval = 0.004[/quote]
sample available here: http://www.ff123.net/samples.html
[quote]Original vs campestre396q0.wav
    8 out of 8, pval = 0.004
Original vs campestre_396b1.wav
    8 out of 8, pval = 0.004
Original vs campestre_3902.wav
    8 out of 8, pval = 0.004[/quote]
uploaded http://xoomer.virgilio.it/fofobella/campestre.flac
[quote]Original vs emmtop_3902.wav
    8 out of 8, pval = 0.004
Original vs emmtop_396b1.wav
    8 out of 8, pval = 0.004
Original vs emmtop396q0.wav
    14 out of 15, pval < 0.001[/quote]
sorry, sample exceeds 30 sec.
[quote]Original vs fall_3902.wav
    16 out of 28, pval = 0.286
Original vs fall_396b1.wav
    8 out of 8, pval = 0.004
Original vs fall396q0.wav
    18 out of 21, pval < 0.001[/quote]
sample uploaded
[quote] Original vs iron396q0.wav
    8 out of 8, pval = 0.004
Original vs iron_396b1.wav
    8 out of 8, pval = 0.004
Original vs iron_3902.wav
    8 out of 8, pval = 0.004[/quote]
available here: http://www.mp3dev.org/mp3/gpsycho/quality.html
[quote]Original vs Rebel396q0.wav
    8 out of 8, pval = 0.004
Original vs Rebel_3902.wav
    8 out of 8, pval = 0.004
Original vs Rebel_396b1.wav
    13 out of 14, pval < 0.001[/quote]
sample uploaded
[quote] Original vs thewayitis_3902.wav
    8 out of 8, pval = 0.004
Original vs thewayitis396q0.wav
    8 out of 8, pval = 0.004
Original vs thewayitis_396b1.wav
    8 out of 8, pval = 0.004[/quote]
available here: http://www.ff123.net/samples.html
[quote]Original vs Tosca_396b1.wav
    8 out of 8, pval = 0.004
Original vs Tosca_3902.wav
    8 out of 8, pval = 0.004
Original vs Tosca396q0.wav
    8 out of 8, pval = 0.004[/quote]
sample uploaded
[quote]Original vs velvet_3902.wav
    8 out of 8, pval = 0.004
Original vs velvet_396b1.wav
    8 out of 8, pval = 0.004
Original vs velvet396q0.wav
    8 out of 8, pval = 0.004[/quote]
available here: http://www.mp3dev.org/mp3/gpsycho/quality.html


--------------------
WavPack 4.3 -mfx5
LAME 3.97 -V5 --vbr-new --athaa-sensitivity 1
Go to the top of the page
+Quote Post
LoFiYo
post Mar 21 2004, 19:30
Post #19





Group: Members
Posts: 133
Joined: 2-January 04
Member No.: 10896



Ticket to Ride by The Beatles

I had to go down to 96kbps, because this one is easy to encode.

lame-3.90.3 --alt-preset 96
vs
lame-3.96b1 --preset 96

QUOTE
ABC/HR Version 0.9b, 30 August 2002
Testname: TICKET TO RIDE - ABR96 - 3.90.3 VS 3.96B1

1L = C:\My Music\lab\tkt\abr96-396b1.mp3.wav
2L = C:\My Music\lab\tkt\abr96-3903.mp3.wav

---------------------------------------
General Comments:
#2 was much harder to ABX than #1. For this sample, #2 is the clear winner.
---------------------------------------
1L File: C:\My Music\lab\tkt\abr96-396b1.mp3.wav
1L Rating: 2.5
1L Comment: The tambourine sounds distorted. The drum roll is also distroted.
---------------------------------------
2L File: C:\My Music\lab\tkt\abr96-3903.mp3.wav
2L Rating: 4.0
2L Comment: The tambourine sounds fine. The drum roll is distored.
---------------------------------------
ABX Results:
Original vs C:\My Music\lab\tkt\abr96-396b1.mp3.wav
    10 out of 11, pval = 0.006
Original vs C:\My Music\lab\tkt\abr96-3903.mp3.wav
    10 out of 13, pval = 0.046
C:\My Music\lab\tkt\abr96-396b1.mp3.wav vs C:\My Music\lab\tkt\abr96-3903.mp3.wav
    5 out of 5, pval = 0.031
Go to the top of the page
+Quote Post
PVNC
post Mar 21 2004, 20:59
Post #20





Group: Members
Posts: 16
Joined: 8-February 04
From: Ottawa, Canada
Member No.: 11844



Sound Blaster Live! 5.1
foobar2000 ABX comparator
DSP: volume control, Resampler (SSRC) 48000Hz
all files replaygained on track basis
Harman/Kardon HK-395 speakers


(-) Ions --alt-preset 128

wav vs 3.90.3 modified (494 KB)
8/8
0.4%
Comment: mp3 version sounds less sharp when volume of buzzing reaches peak level; mp3 sounds kind of smeared.

wav vs 3.96b1 (520 KB)
8/8
0.4%
Comment: distortions between volume peaks - this affects the lower-volume buzzing; (peak volume buzzing seems to sound like original wav).

3.90.3 modified (494 KB) vs 3.96b1 (520 KB)
8/8
0.4%
Comment: harder to ABX the two encoded versions. I found that 3.96's distortions were easier to differentiate the two than the peak buzzing volume. 3.90.3 is harder to pick out, at least with my ears.


Rock & Roll --alt-preset 128

wav vs 3.90.3 modified (442 KB)
8/8
0.4%
Comment: cymbals and drum solo sound very "smeared." Also I noticed what seemed like an instant of stereo collapse.

wav vs 3.96b1 (449 KB)
8/8
0.4%
Comment: similar artifacts as 3.90.3, with the initial part of the drum solo seeming to suffer from a HF boost, but I did not notice the stereo problem.

3.90.3 modified (442 KB) vs 3.96b1 (449 KB)
17/21
(ABX comparator gives me N/A as a probability)
Comment: This was the toughest. The smearing of cymbals was of no use to differentiate the two, as it occurs in both, and I stopped getting positive results with the supposed HF boost in the drum solo (hence the 4 failed trials). Then I focused on the end of the last chorus and the final vocal, which sounded more distorted in 3.90.3 than in 3.96. That turned out to be the main difference.
Go to the top of the page
+Quote Post
harashin
post Mar 22 2004, 08:42
Post #21





Group: Members
Posts: 339
Joined: 20-February 02
From: Kyoto, Japan
Member No.: 1362



I've done some tests to verify results of other members.

death2 myf_4sec doesnair


--------------------
Folding@Home Hydrogenaudio.org Team ID# 32639
http://folding.stanford.edu/
Go to the top of the page
+Quote Post
owowo
post Mar 23 2004, 03:14
Post #22





Group: Members
Posts: 7
Joined: 6-March 04
Member No.: 12526



Sample used: Lazy_Jones.flac
TEST: 3.96b1 (--preset extreme) <287.5 Kbps> vs. 3.90.3 (--alt-preset extreme) <273.9 Kbps>
Average bitrates according to Nero6
Encoded with lame.exe/LAMEDrop1.3 (l3maniac) - Decoded with Nero6
CODE
-------------------------------------
WinABX v0.42 test report
03/22/2004 20:50:33

A file: C:\WINDOWS\Desktop\Lazy_Jones (wav).wav
B file: C:\WINDOWS\Desktop\Lazy_Jones 3.90.3 (--ap e).wav

20:50:55    1/1  p=50.0%
20:51:02    2/2  p=25.0%
20:51:11    3/3  p=12.5%
20:51:19    4/4  p=6.2%
20:51:34    5/5  p=3.1%
20:51:42    6/6  p=1.6%
20:51:50    7/7  p=0.8%
20:51:59    8/8  p=0.4%
20:52:09    9/9  p=0.2%
20:52:10  reset

20:52:19  test finished

-------------------------------------
WinABX v0.42 test report
03/22/2004 20:52:19

A file: C:\WINDOWS\Desktop\Lazy_Jones (wav).wav
B file: C:\WINDOWS\Desktop\Lazy_Jones 3.96b1 (--p e).wav

20:52:32    1/1  p=50.0%
20:52:45    2/2  p=25.0%
20:52:58    3/3  p=12.5%
20:53:09    4/4  p=6.2%
20:53:17    5/5  p=3.1%
20:53:26    6/6  p=1.6%
20:53:34    7/7  p=0.8%
20:53:44    8/8  p=0.4%
20:53:53    9/9  p=0.2%
20:53:54  reset

20:54:01  test finished

-------------------------------------
WinABX v0.42 test report
03/22/2004 20:54:01

A file: C:\WINDOWS\Desktop\Lazy_Jones 3.90.3 (--ap e).wav
B file: C:\WINDOWS\Desktop\Lazy_Jones 3.96b1 (--p e).wav

20:54:12    1/1  p=50.0%
20:54:20    2/2  p=25.0%
20:54:28    3/3  p=12.5%
20:54:36    4/4  p=6.2%
20:54:44    5/5  p=3.1%
20:54:52    6/6  p=1.6%
20:55:01    7/7  p=0.8%
20:55:09    8/8  p=0.4%
20:55:19    9/9  p=0.2%
20:55:21  reset

20:55:23  test finished

Comments:
3.90.3 (--alt-preset extreme); artifacts, sound like 'hi-hat close'
3.96b1 (--preset extreme); fatal flaw during initial attack during the first few milliseconds, sounds similar to a 'missing frame header' error, dead give-away not present in 3.90.3. Much clearer than 3.90.3, 'hi-hat close' artifacts not nearly as audible, but still present.
_____________________
Sample used: Lazy_Jones.flac
TEST: 3.96b1 (--preset insane) vs. 3.90.3 (--alt-preset insane)
Encoded with lame.exe/LAMEDrop1.3 (l3maniac) - Decoded with Nero6
CODE
-------------------------------------
WinABX v0.42 test report
03/22/2004 20:43:52

A file: C:\WINDOWS\Desktop\Lazy_Jones (wav).wav
B file: C:\WINDOWS\Desktop\Lazy_Jones 3.90.3 (--ap i).wav

20:44:34    1/1  p=50.0%
20:44:43    2/2  p=25.0%
20:44:54    3/3  p=12.5%
20:45:03    4/4  p=6.2%
20:45:12    5/5  p=3.1%
20:45:21    6/6  p=1.6%
20:45:29    7/7  p=0.8%
20:45:38    8/8  p=0.4%
20:45:46    9/9  p=0.2%
20:45:50  reset

20:45:59  test finished

-------------------------------------
WinABX v0.42 test report
03/22/2004 20:45:59

A file: C:\WINDOWS\Desktop\Lazy_Jones (wav).wav
B file: C:\WINDOWS\Desktop\Lazy_Jones 3.96b1 (--p i).wav

Start position 00:00.0, end position 00:06.3
20:46:30    1/1  p=50.0%
20:46:41    2/2  p=25.0%
20:46:51    3/3  p=12.5%
20:46:59    4/4  p=6.2%
20:47:08    5/5  p=3.1%
20:47:17    6/6  p=1.6%
20:47:26    7/7  p=0.8%
20:47:35    8/8  p=0.4%
20:47:45    9/9  p=0.2%
20:47:48  reset

20:48:00  test finished

-------------------------------------
WinABX v0.42 test report
03/22/2004 20:48:00

A file: C:\WINDOWS\Desktop\Lazy_Jones 3.90.3 (--ap i).wav
B file: C:\WINDOWS\Desktop\Lazy_Jones 3.96b1 (--p i).wav

20:48:17    1/1  p=50.0%
20:48:25    2/2  p=25.0%
20:48:34    3/3  p=12.5%
20:48:42    4/4  p=6.2%
20:48:51    5/5  p=3.1%
20:48:59    6/6  p=1.6%
20:49:09    7/7  p=0.8%
20:49:17    8/8  p=0.4%
20:49:27    9/9  p=0.2%
20:49:29  reset

20:49:31  test finished

Comments:
3.90.3 (--alt-preset insane); faint artifacts sound like "hi-hat close"
3.96b1 (--preset insane); fatal flaw during initial attack during the first few milliseconds, sounds similar to a 'missing frame header' error, dead give-away not present in 3.90.3. Much learer than 3.90.3
Go to the top of the page
+Quote Post
freakngoat
post Mar 25 2004, 05:04
Post #23





Group: Members
Posts: 170
Joined: 10-January 02
From: Manhattan Beach
Member No.: 979



I am verifying Lazy Jones 3.90.3 aps vs. 3.96b1 ps. I focused on the last 1.5s of the sample:

3.90.3 aps vs. original
8/8 - popping artifacts in the last second of sample, very obvious

3.96b1 aps vs original
unable to ABX - sounds perfect

3.90.3 aps vs. 3.96b1 aps
8/8

edit: unable to ABX 3.96b1 or 3.90.3 at ape

This post has been edited by freakngoat: Mar 25 2004, 05:14
Go to the top of the page
+Quote Post
harashin
post Mar 25 2004, 07:07
Post #24





Group: Members
Posts: 339
Joined: 20-February 02
From: Kyoto, Japan
Member No.: 1362



I prefer 3.90.3 --alt-preset standard to 3.96b1 --preset standard on these samples.
Destitute Rosemary


--------------------
Folding@Home Hydrogenaudio.org Team ID# 32639
http://folding.stanford.edu/
Go to the top of the page
+Quote Post
freakngoat
post Mar 25 2004, 09:14
Post #25





Group: Members
Posts: 170
Joined: 10-January 02
From: Manhattan Beach
Member No.: 979



I tested drone_short, 0-1 second:

3.90.3 aps vs. original 0-1s
8/8 - sounds like air being blown

3.96b1 aps vs. original 0-1s
8/8 - much easier to ABX, air sound louder and more noticable

3.96b1 aps vs. 3.90.3 aps
7/8 - missed last test, ears getting fatigued

3.90.3 sounds better

This post has been edited by freakngoat: Mar 25 2004, 09:31
Go to the top of the page
+Quote Post

3 Pages V   1 2 3 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 17th September 2014 - 17:45