IPB

Welcome Guest ( Log In | Register )

3 Pages V  < 1 2 3  
Reply to this topicStart new topic
LAME 3.96 vs. 3.90.3 Test, Test/Result Thread
tigre
post Apr 9 2004, 16:44
Post #51


Moderator


Group: Members
Posts: 1434
Joined: 26-November 02
Member No.: 3890



Sample : Its me ~ 128kbps tested

Live recording, focussed on problems with applause, details see ABC/HR log
3.96b2 -V 5 won (3.0)
3.90.3 ap 128 was second (2.0)
3.96b2 p 128 was worst (1.5)

ABC/HR log (I tested vorbis codecs with HF reduction as well, to save space here I cut out the related parts in ABC/HR log.)
QUOTE
ABC/HR Version 0.9b, 30 August 2002
Testname:

1L = N:\lametest\Soul_Live\decode\017 Its me.wav_mtb3_4.wav
2L = N:\lametest\Soul_Live\decode\011 Its me.wav_23_4.wav
3L = N:\lametest\Soul_Live\decode\012 Its me.wav_3903_ap128.wav
4R = N:\lametest\Soul_Live\decode\015 Its me.wav_cqk32_4.25.wav
5R = N:\lametest\Soul_Live\decode\010 Its me.wav.wav
6R = N:\lametest\Soul_Live\decode\016 Its me.wav_hfr_4.25.wav
7R = N:\lametest\Soul_Live\decode\014 Its me.wav_396b2_V5.wav
8R = N:\lametest\Soul_Live\decode\013 Its me.wav_396b2_p128.wav

---------------------------------------
General Comments:

---------------------------------------
<snip>
---------------------------------------
3L File: N:\lametest\Soul_Live\decode\012 Its me.wav_3903_ap128.wav
3L Rating: 2.0
3L Comment: 0.0-3.4: surround-like, warbeling/metallic sound in applause, no hf noise boost -> should be lame.
---------------------------------------
<snip>
---------------------------------------
7R File: N:\lametest\Soul_Live\decode\014 Its me.wav_396b2_V5.wav
7R Rating: 3.0
7R Comment: should be lame; small ringing on right channel, otherwise quite good
---------------------------------------
8R File: N:\lametest\Soul_Live\decode\013 Its me.wav_396b2_p128.wav
8R Rating: 1.5
8R Comment: sounds like 3 with hf ringing problems of 7 added
---------------------------------------
ABX Results:
<snip>
Original vs N:\lametest\Soul_Live\decode\012 Its me.wav_3903_ap128.wav
    7 out of 7, pval = 0.008
<snip>
Original vs N:\lametest\Soul_Live\decode\014 Its me.wav_396b2_V5.wav
    7 out of 7, pval = 0.008
Original vs N:\lametest\Soul_Live\decode\013 Its me.wav_396b2_p128.wav
    7 out of 7, pval = 0.008
<snip>
N:\lametest\Soul_Live\decode\012 Its me.wav_3903_ap128.wav vs N:\lametest\Soul_Live\decode\014 Its me.wav_396b2_V5.wav
    11 out of 12, pval = 0.003
N:\lametest\Soul_Live\decode\012 Its me.wav_3903_ap128.wav vs N:\lametest\Soul_Live\decode\013 Its me.wav_396b2_p128.wav
    11 out of 12, pval = 0.003
<snip>
N:\lametest\Soul_Live\decode\014 Its me.wav_396b2_V5.wav vs N:\lametest\Soul_Live\decode\013 Its me.wav_396b2_p128.wav
    7 out of 7, pval = 0.008


--------------------
Let's suppose that rain washes out a picnic. Who is feeling negative? The rain? Or YOU? What's causing the negative feeling? The rain or your reaction? - Anthony De Mello
Go to the top of the page
+Quote Post
tigre
post Apr 9 2004, 16:52
Post #52


Moderator


Group: Members
Posts: 1434
Joined: 26-November 02
Member No.: 3890



Sample : entierren con rumba ~ 128kbps tested

Salsa with lots of percussion, details see ABC/HR log, results similar to the previous sample I tested
3.96b2 -V 5 won
3.90.3 ap 128 was second
3.96b2 p 128 was worst

ABC/HR log:
QUOTE
ABC/HR Version 0.9b, 30 August 2002
Testname: entierren con rumba lame 128

1L = N:\lametest\Salsa\decode\entierren con rumba.wav_396b2_V5_.wav
2L = N:\lametest\Salsa\decode\entierren con rumba.wav_396b2_p128_.wav
3L = N:\lametest\Salsa\decode\entierren con rumba.wav_3903_ap128_.wav

---------------------------------------
General Comments:
Guiros (percussion) smeared, clearly noticable (and ABXed) at 2.8-6.0
1 is best, 2 and 3 are similar but 2 has ringing/metallic sound additionally.
After 6.0 it's similar with the additonal precussion instrument (shaker?).
---------------------------------------
1L File: N:\lametest\Salsa\decode\entierren con rumba.wav_396b2_V5_.wav
1L Rating: 3.5
1L Comment:
---------------------------------------
2L File: N:\lametest\Salsa\decode\entierren con rumba.wav_396b2_p128_.wav
2L Rating: 2.0
2L Comment:
---------------------------------------
3L File: N:\lametest\Salsa\decode\entierren con rumba.wav_3903_ap128_.wav
3L Rating: 2.5
3L Comment:
---------------------------------------
ABX Results:
Original vs N:\lametest\Salsa\decode\entierren con rumba.wav_396b2_V5_.wav
    7 out of 7, pval = 0.008
Original vs N:\lametest\Salsa\decode\entierren con rumba.wav_396b2_p128_.wav
    7 out of 7, pval = 0.008
Original vs N:\lametest\Salsa\decode\entierren con rumba.wav_3903_ap128_.wav
    7 out of 7, pval = 0.008
N:\lametest\Salsa\decode\entierren con rumba.wav_396b2_V5_.wav vs N:\lametest\Salsa\decode\entierren con rumba.wav_396b2_p128_.wav
    7 out of 7, pval = 0.008
N:\lametest\Salsa\decode\entierren con rumba.wav_396b2_V5_.wav vs N:\lametest\Salsa\decode\entierren con rumba.wav_3903_ap128_.wav
    7 out of 7, pval = 0.008
N:\lametest\Salsa\decode\entierren con rumba.wav_396b2_p128_.wav vs N:\lametest\Salsa\decode\entierren con rumba.wav_3903_ap128_.wav
    7 out of 7, pval = 0.008



____________________________________________________________________


BTW: IT WOULD BE GREAT IF SOME MORE PEOPLE COULD PROVIDE ~128KBPS RESULTS BEFORE RJAMORIM'S 128KBPS TEST STARTS.


--------------------
Let's suppose that rain washes out a picnic. Who is feeling negative? The rain? Or YOU? What's causing the negative feeling? The rain or your reaction? - Anthony De Mello
Go to the top of the page
+Quote Post
ViPER1313
post Apr 11 2004, 21:29
Post #53





Group: Members
Posts: 271
Joined: 19-August 02
From: Maryland
Member No.: 3109



Lame v3.96 final seems to have a serious problem w/ 41_30sec in the 6-8 second range using --preset cbr 128. It sounds like a dropout / skip in the sound. This problem has been seen in Mitok's compile as well as ICL v8.0 and MSVC compiles made by myself. There is no skip using v3.90.3 . Could someone confirm this??

EDIT - Look at the waveforms - the volumes between the wav/3.90.3 and 3.96 are drastically different.

EDIT2 - Lame v3.96b2 does not suffer from this problem either.

Origional WAV:



Lame v3.90.3 --preset cbr 128:



Lame v3.96 --preset cbr 128:



This post has been edited by ViPER1313: Oct 20 2006, 09:22
Go to the top of the page
+Quote Post
SometimesWarrior
post Apr 12 2004, 00:33
Post #54





Group: Members
Posts: 671
Joined: 21-November 01
From: California, US
Member No.: 514



QUOTE (ViPER1313 @ Apr 11 2004, 12:29 PM)
Lame v3.96 final seems to have a serious problem w/ 41_30sec in the 6-8 second range using --preset cbr 128. It sounds like a dropout / skip in the sound. This problem has been seen in Mitok's compile as well as ICL v8.0 and MSVC compiles made by myself. There is no skip using v3.90.3 . Could someone confirm this??

I can confirm it. I tested the 4.5-8.5 second range of 41_30sec, using Mitiok's 3.96 and john33's 3.90.3 and 3.96b2. The 3.96 final version definitely has some nasty dropouts not present the other encodes.

CODE
ABC/HR Version 0.9b, 30 August 2002
Testname:

1L = G:\Filetemp\diskwriter\41_30\41-396.wav
2L = G:\Filetemp\diskwriter\41_30\41-3903.wav
3R = G:\Filetemp\diskwriter\41_30\41-396b2.wav

---------------------------------------
General Comments:

---------------------------------------
1L File: G:\Filetemp\diskwriter\41_30\41-396.wav
1L Rating: 1.0
1L Comment: Pops or dropouts
---------------------------------------
2L File: G:\Filetemp\diskwriter\41_30\41-3903.wav
2L Rating: 2.5
2L Comment:
---------------------------------------
3R File: G:\Filetemp\diskwriter\41_30\41-396b2.wav
3R Rating: 2.2
3R Comment: A bit more swirling than sample 2
---------------------------------------
ABX Results:
G:\Filetemp\diskwriter\41_30\41-396.wav vs G:\Filetemp\diskwriter\41_30\41-3903.wav
   8 out of 8, pval = 0.004
G:\Filetemp\diskwriter\41_30\41-396.wav vs G:\Filetemp\diskwriter\41_30\41-396b2.wav
   8 out of 8, pval = 0.004
G:\Filetemp\diskwriter\41_30\41-3903.wav vs G:\Filetemp\diskwriter\41_30\41-396b2.wav
   11 out of 12, pval = 0.003


This post has been edited by SometimesWarrior: Apr 12 2004, 00:35
Go to the top of the page
+Quote Post
Brainchild4010
post Apr 12 2004, 10:13
Post #55





Group: Members
Posts: 1
Joined: 12-April 04
Member No.: 13432



crying.gif Yes , i also can confirm it.

I test it too with 41_30sec file and at the 4.5 - 6 is definitely a Pop with Lame 3.96 Final.

No Problems with Lame 3.90.3 and 3.96b2.
Go to the top of the page
+Quote Post
ViPER1313
post Apr 12 2004, 23:20
Post #56





Group: Members
Posts: 271
Joined: 19-August 02
From: Maryland
Member No.: 3109



I have created a gif animation that shows how the volume levels seem to change / drop out in the v3.96 final encoder when using 128kbps and to a lesser extent 160kbps - it can be found here. Take a careful look at v3.90.3 vs. v3.96. More testing on this issue and bitrates compromised by it need to be performed.

This is just a random section (1.6-1.8sec) of the 41_30 clip, and this section is ABX-able (sounds like a click / popping).

This post has been edited by ViPER1313: Apr 12 2004, 23:20
Go to the top of the page
+Quote Post
ff123
post Apr 12 2004, 23:29
Post #57


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



QUOTE (ViPER1313 @ Apr 12 2004, 02:20 PM)
I have created a gif animation that shows how the volume levels seem to change / drop out in the v3.96 final encoder when using 128kbps and to a lesser extent 160kbps - it can be found  here. Take a careful look at v3.90.3 vs. v3.96. More testing on this issue and bitrates compromised by it need to be performed.

This is just a random section (1.6-1.8sec) of the 41_30 clip, and this section is ABX-able (sounds like a click / popping).

Looks like a bug. Does this occur with abr as well (p 128)?

That could be a reason to eliminate 3.96final from consideration in Roberto's upcoming multiformat test.

ff123
Go to the top of the page
+Quote Post
ViPER1313
post Apr 12 2004, 23:36
Post #58





Group: Members
Posts: 271
Joined: 19-August 02
From: Maryland
Member No.: 3109



From what I have seen and heard it does not seem to occur in ABR or VBR modes, although the encoder could just be throwing bits to where they are needed. The problems shown in 41_30sec seem to disappear using --preset 128, so IMO ABR seems fine.

This post has been edited by ViPER1313: Apr 12 2004, 23:38
Go to the top of the page
+Quote Post
Fr4nz
post Apr 13 2004, 00:25
Post #59





Group: Members
Posts: 102
Joined: 30-June 03
From: Venice (yeah, city of the gondola!!)
Member No.: 7470



OMG! At www.cdfreaks.com have published this new Lame version! Someone should told them to remove the news because of this buggy version.
Go to the top of the page
+Quote Post
damiandimitri
post Apr 13 2004, 06:06
Post #60





Group: Members
Posts: 11
Joined: 6-November 02
Member No.: 3710



thank you for the warning.....

i will try to contact an operator at cdfreaks
Go to the top of the page
+Quote Post
Debro
post Apr 13 2004, 07:54
Post #61





Group: Members
Posts: 1
Joined: 13-April 04
Member No.: 13455



Cdfreaks has a disclaimer at the bottom of the article warning users of the bug smile.gif
Go to the top of the page
+Quote Post
Gabriel
post Apr 13 2004, 08:58
Post #62


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



I would like people testing a little more than 1 sample before calling a version "buggy"

edit: additionnaly it could help if you could find other clearly problematic samples where you hear a similar behaviour (problem in cbr but not in abr)

This post has been edited by Gabriel: Apr 13 2004, 10:55
Go to the top of the page
+Quote Post
funkyblue
post Apr 13 2004, 10:03
Post #63





Group: Members
Posts: 322
Joined: 28-November 01
From: South Australia
Member No.: 555



I agree with Gabriel, who is to say that Lame 3.96 may break one sample, but fix 25 other samples? Unless it is tested more, you can't judge

This post has been edited by burgerings: Apr 13 2004, 10:05
Go to the top of the page
+Quote Post
Vietwoojagig
post Apr 13 2004, 12:30
Post #64





Group: Members
Posts: 248
Joined: 28-November 02
From: Germany, Trier
Member No.: 3916



QUOTE (Gabriel @ Apr 13 2004, 08:58 AM)
I would like people testing a little more than 1 sample before calling a version "buggy"

edit: additionnaly it could help if you could find other clearly problematic samples where you hear a similar behaviour (problem in cbr but not in abr)

But, on the other hand, it would be good to know, why this sample is problematic with 3.96 and not with 3.96b2. Do you have any guess?
Go to the top of the page
+Quote Post
ViPER1313
post Apr 13 2004, 15:06
Post #65





Group: Members
Posts: 271
Joined: 19-August 02
From: Maryland
Member No.: 3109



Tested "Waiting" sample at --preset cbr 128 using Lame v3.90.3, 3.96b2 and v3.96 final. Tested in the 0-2.2 second region of the song. While there are no clicks, there seems to be a pretty major quality regression between v3.96b2 and v3.96 final, while 3.90.3 still comes out as the winner.

QUOTE
ABC/HR Version 0.9b, 30 August 2002
Testname: Waiting --preset cbr 128

1L = Z:\Music\Temp\3.96 Final.wav
2R = Z:\Music\Temp\3.90.3.wav
3L = Z:\Music\Temp\v3.96b2.wav

---------------------------------------
General Comments:

---------------------------------------
1L File: Z:\Music\Temp\3.96 Final.wav
1L Rating: 3.7
1L Comment: Horrible warble / distortion in the 0-2 second region
---------------------------------------
2R File: Z:\Music\Temp\3.90.3.wav
2R Rating: 4.5
2R Comment: Best of the 3, very clean, no major distortions.
---------------------------------------
3L File: Z:\Music\Temp\v3.96b2.wav
3L Rating: 4.3
3L Comment: Slightly worse than 2, much better than 1, slight warble on the beginning pronunciation of "been."
---------------------------------------
ABX Results:
Original vs Z:\Music\Temp\3.96 Final.wav
    8 out of 8, pval = 0.004
Original vs Z:\Music\Temp\3.90.3.wav
    8 out of 8, pval = 0.004
Original vs Z:\Music\Temp\v3.96b2.wav
    8 out of 8, pval = 0.004
Z:\Music\Temp\3.96 Final.wav vs Z:\Music\Temp\3.90.3.wav
    13 out of 14, pval < 0.001
Z:\Music\Temp\3.96 Final.wav vs Z:\Music\Temp\v3.96b2.wav
    16 out of 17, pval < 0.001
Z:\Music\Temp\3.90.3.wav vs Z:\Music\Temp\v3.96b2.wav
    18 out of 20, pval < 0.001
Go to the top of the page
+Quote Post
Gabriel
post Apr 13 2004, 15:16
Post #66


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



QUOTE
Tested "Waiting" sample at --preset cbr 128 using Lame v3.90.3, 3.96b2 and v3.96 final.

What about abr?
Go to the top of the page
+Quote Post
ViPER1313
post Apr 13 2004, 16:09
Post #67





Group: Members
Posts: 271
Joined: 19-August 02
From: Maryland
Member No.: 3109



QUOTE (Gabriel @ Apr 13 2004, 10:16 AM)
QUOTE
Tested "Waiting" sample at --preset cbr 128 using Lame v3.90.3, 3.96b2 and v3.96 final.

What about abr?

Your wish is my command biggrin.gif - "--preset 128" using v3.90.3, 3.96b2 and 3.96 final.

QUOTE
ABC/HR Version 0.9b, 30 August 2002
Testname: Waiting --preset 128

1R = Z:\Music\Temp\3.96 ABR.wav
2L = Z:\Music\Temp\3.90.3 ABR.wav
3L = Z:\Music\Temp\3.96b2 ABR.wav

---------------------------------------
General Comments:
All samples did better than in CBR, and no codec sounded horrible.
---------------------------------------
1R File: Z:\Music\Temp\3.96 ABR.wav
1R Rating: 4.1
1R Comment: Felt this had slightly more warble than 3 and is clearly worse than 2, although  I am not 100% sure that I can ABX it against 3.
---------------------------------------
2L File: Z:\Music\Temp\3.90.3 ABR.wav
2L Rating: 4.6
2L Comment: Better than 1 or 3, sounded cleaner
---------------------------------------
3L File: Z:\Music\Temp\3.96b2 ABR.wav
3L Rating: 4.2
3L Comment: Slight warble, thought it was slightly better than 1 but could not ABX it with 100% confidence - got a 15 of  17, p = 0.001 but with later trials could not ABX - might just be listening fatigue.
---------------------------------------
ABX Results:
Original vs Z:\Music\Temp\3.96 ABR.wav
    8 out of 8, pval = 0.004
Original vs Z:\Music\Temp\3.90.3 ABR.wav
    8 out of 8, pval = 0.004
Original vs Z:\Music\Temp\3.96b2 ABR.wav
    13 out of 14, pval < 0.001
Z:\Music\Temp\3.96 ABR.wav vs Z:\Music\Temp\3.90.3 ABR.wav
    13 out of 14, pval < 0.001
Z:\Music\Temp\3.96 ABR.wav vs Z:\Music\Temp\3.96b2 ABR.wav
    28 out of 42, pval = 0.022
Z:\Music\Temp\3.90.3 ABR.wav vs Z:\Music\Temp\3.96b2 ABR.wav
    10 out of 10, pval < 0.001
Go to the top of the page
+Quote Post
ViPER1313
post Apr 13 2004, 16:24
Post #68





Group: Members
Posts: 271
Joined: 19-August 02
From: Maryland
Member No.: 3109



At this point in time, I would sum up the whole situation by saying:
  • I don't know if 41_30sec is an isolated case or not - I agree that more samples are needed before any judgment can be made.
  • I have only seen the problem in --preset cbr 128 - other bitrates/modes donít exhibit the problem in the same way or only to a degree that cannot be easily heard.
  • I would not go so far as to call v3.96 buggy, but more testing needs to be done on the final version.
  • v3.90.3 still seems to have higher quality in most samples at most bitrates (with the exception of fatboy and a few others)
  • MORE TESTING OF v3.96 IS NEEDED
If other people could do some tests to prove or disprove what I have found, it would be a large help.
Go to the top of the page
+Quote Post
Gabriel
post Apr 13 2004, 16:33
Post #69


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



QUOTE
I don't know if 41_30sec is an isolated case or not - I agree that more samples are needed before any judgment can be made.


I can confirm that there is a failure on this sample on frame 272, 2nd granule (7.11s) when encoding in cbr 128. The output on mp3X is very strange.
Go to the top of the page
+Quote Post
tigre
post Apr 13 2004, 17:21
Post #70


Moderator


Group: Members
Posts: 1434
Joined: 26-November 02
Member No.: 3890



OK. I've started a LAME 3.96 FINAL vs. 3.90.3 Test, Test/Result Thread.

I'm a bit short of time right now - I'll add missing results to the related posts ASAP if noone's faster.


Gabriel:
QUOTE
I can confirm that there is a failure on this sample on frame 272, 2nd granule (7.11s) when encoding in cbr 128. The output on mp3X is very strange.

Is this something that is going to be fixed soon (so we can expect 3.96.1 or similar in a couple of days and should better wait with testing 3.96 final) or is it just the price that has to be paid for better performance elsewhere?

-----------------------------------------------------------------
DJED's post split to a separate thread.


--------------------
Let's suppose that rain washes out a picnic. Who is feeling negative? The rain? Or YOU? What's causing the negative feeling? The rain or your reaction? - Anthony De Mello
Go to the top of the page
+Quote Post
Gabriel
post Apr 13 2004, 18:07
Post #71


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



QUOTE
Is this something that is going to be fixed soon (so we can expect 3.96.1 or similar in a couple of days and should better wait with testing 3.96 final) or is it just the price that has to be paid for better performance elsewhere?


Well, I think that this specific one should be fixed, but I do not know how soon this will be done.
My opinion is that you can test 3.96.
Go to the top of the page
+Quote Post
tigre
post Apr 14 2004, 12:02
Post #72


Moderator


Group: Members
Posts: 1434
Joined: 26-November 02
Member No.: 3890



I've just edited all relevant 3.96beta results that were still missing in the results post.

For further testing of 3.96 final as well as for continuing related discussion go here please.

If necessary I can start a separate thread for 3.96beta -> final regression cases and move related posts there. Opinions please?


--------------------
Let's suppose that rain washes out a picnic. Who is feeling negative? The rain? Or YOU? What's causing the negative feeling? The rain or your reaction? - Anthony De Mello
Go to the top of the page
+Quote Post

3 Pages V  < 1 2 3
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 26th October 2014 - 05:55