IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
lame 3.90.3 vs lame 3.96.1 at ~130 kbps, ABR and VBR
guruboolez
post Nov 15 2004, 09:37
Post #1





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



Interested by performance of various lame encoders/settings at ~130 kbps (I'm using a small flash memory player), and also bored by unproductive quarrel about this question, I've decided to test them myself.
The tests consist on simple ABCHR ratings, without ABX confrontations. When two (or three) files sounded very similar, I didn’t insist, and gave both the same note. ANOVA and TUKEY analysis will conclude the test.

• SAMPLES

The 18 samples used in last Roberto's test. In other words, it’s a wide panel of musical genre. I'm more interested in “classical” music, but this musical genre includes too many situations, and therefore need a dedicated test to be really useful. Maybe I’ll do it later…

• CHALLENGERS

— The "tested" 3.90.3 with --alt-preset 130 [ABR 130 kbps]
— The "untested" 3.96.1 with --preset 130 [ABR 130 kbps]
— The ignored 3.96.1 -V 5 --athaa-sensitivity 1 [VBR ~130 kbps]

The average bitrate of 3.96.1 VBR mode is 129 kbps, and extreme values are 89 kbps and 153 kbps.

• RESULTS




• ANALYSIS (basic)

Results are really contrasted: lame 3.96.1 VBR appears as the big winner (1st on 72% of tested samples), and 3.96.1 ABR as the worse (last on 72% of total samples). The old lame release sounded better than 3.96.1 ABR/VBR on one sample only [getiton.wav], but on the other side was never ranked as last. Important thing to note: there are much more differences between 3.90.3 ABR and 3.96.1 VBR than between 3.90.3 ABR and 3.96.1 ABR!

• ANALYSIS (advanced)

Using ANOVA tool first and then TUKEY¨PARAMETRIC analysis:

ANOVA ANALYSIS:
CODE
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 18
Critical significance:  0.05
Significance of data: 1.32E-004 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total               53          67.05
Testers (blocks)    17          42.65
Codecs eval'd        2           9.97    4.99   11.75  1.32E-004
Error               34          14.43    0.42
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.441

Means:

3.96.V   3.90.A   3.96.A  
 3.48     2.82     2.44  

---------------------------- p-value Matrix ---------------------------

        3.90.A   3.96.A  
3.96.V   0.004*   0.000*  
3.90.A            0.096    
-----------------------------------------------------------------------

3.96.V is better than 3.90.A, 3.96.A


PARAMETRIC TUKEY ANALYSIS

CODE
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Tukey HSD analysis

Number of listeners: 18
Critical significance:  0.05
Tukey's HSD:   0.533

Means:

3.96.V   3.90.A   3.96.A  
 3.48     2.82     2.44  

-------------------------- Difference Matrix --------------------------

        3.90.A   3.96.A  
3.96.V     0.667*   1.039*
3.90.A              0.372  
-----------------------------------------------------------------------

3.96.V is better than 3.90.A, 3.96.A


In both cases, lame 3.96.1 VBR is said better than lame ABR (both 3.90.3 and 3.96.1). Difference between 3.90.3 ABR and 3.96.1 is not significant enough to conclude with confidence on lame 3.90.3 superiority.


Conclusion: for ~130 kbps encodings, lame 3.96.1 is better (for me) than 3.90.3


P.S. ABC/HR logs are available here

This post has been edited by guruboolez: Dec 29 2005, 21:40
Go to the top of the page
+Quote Post
[proxima]
post Nov 15 2004, 11:01
Post #2





Group: Members
Posts: 197
Joined: 12-October 02
From: Italy
Member No.: 3537



QUOTE (guruboolez @ Nov 15 2004, 09:37 AM)
— The "tested" 3.90.3 with --alt-preset 130 [ABR 130 kbps]
— The "untested" 3.96.1 with --preset 130 [ABR 130 kbps]
— The ignored 3.96.1 -V 5 --athaa-sensitivity 1 [VBR ~130 kbps]

laugh.gif
In reality 3.96.1 ABR and VBR have been tested by some people before the last multiformat 128 test when choosing the competitors. Nevertheless, i agree with your results: 3.96.1 -V5 --athaa-sensitivity 1 is better the the others. 3.96.1 ABR is really worse than the past version but Gabriel is working well and i already listened a good 3.97 alpha version. Warbling and HF problems seem resolved wink.gif
QUOTE
P.S. ABC/HR logs are available here

I'm curious about the sample where 3.96.1 VBR received a low score. There are some samples where 3.90.3 is still superior (i noted that too). I perceived a sort of watery/ringing sound that --athaa-sensitivity 1 reduces but not eliminate completely (http://www.hydrogenaudio.org/forums/index....topic=21079&hl=).
What's your motivation for scoring so little ?


--------------------
WavPack 4.3 -mfx5
LAME 3.97 -V5 --vbr-new --athaa-sensitivity 1
Go to the top of the page
+Quote Post
shadowking
post Nov 15 2004, 11:40
Post #3





Group: Members
Posts: 1529
Joined: 31-January 04
Member No.: 11664



Thanks guru !

I also tested some samples with V5 vs. ABR (lame 3.96.1) and my results were the same as yours. VBR is superior for me as well.


--------------------
Wavpack -b450s0.7
Go to the top of the page
+Quote Post
guruboolez
post Nov 15 2004, 14:06
Post #4





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE ([proxima] @ Nov 15 2004, 11:01 AM)
I'm curious about the sample where 3.96.1 VBR received a low score (...)
What's your motivation for scoring so little ?
*


Are you talking about one precise sample, or all problem samples for 3.96.1-VBR?
I didn't use any low anchor -> notation could be exagerated in some case. Nevertheless, when a file was ranked 2.0/5, it means that I was really annoyed by some distortions (not necessary ringing, but also metallic/acid colouration).


P.S. Sorry, I thought I had reply to your question. I don't know what happened...
Go to the top of the page
+Quote Post
dev0
post Nov 15 2004, 17:46
Post #5





Group: Developer
Posts: 1679
Joined: 23-December 01
From: Germany
Member No.: 731



These results confirm what has been stated by [proxima], myself and others before: 3.96.1's CBR/ABR mode has some serious regressions compared to 3.90.3, but Gabriel is working hard on fixing them.
I find the performance of -V5 to be seriously impressive for its bitrate (even though my music tends to bloat up to 150kbps with this setting and is still far from transparent -> see sincealways).


--------------------
"To understand me, you'll have to swallow a world." Or maybe your words.
Go to the top of the page
+Quote Post
Tang
post Nov 15 2004, 18:13
Post #6





Group: Members (Donating)
Posts: 158
Joined: 27-January 04
Member No.: 11536



Hi Guru,
I've just read your results (very interesting as usual, thanks) but i don't understand something... Why didn't you compare the 3.96.1 VBR with modified 3.90.3 "--alt-preset medium"???
This wouldn't have made sense?

Anyway thanks for your test results... (was finally using 3.96.1 for some weeks however)

Best regards smile.gif
Go to the top of the page
+Quote Post
dev0
post Nov 15 2004, 18:57
Post #7





Group: Developer
Posts: 1679
Joined: 23-December 01
From: Germany
Member No.: 731



--alt-preset medium would have come out quite a bit over the target bitrate of 130kbps.


--------------------
"To understand me, you'll have to swallow a world." Or maybe your words.
Go to the top of the page
+Quote Post
Tang
post Nov 15 2004, 21:36
Post #8





Group: Members (Donating)
Posts: 158
Joined: 27-January 04
Member No.: 11536



QUOTE (dev0 @ Nov 15 2004, 09:57 AM)
--alt-preset medium would have come out quite a bit over the target bitrate of 130kbps.
*

Ok I thought -apm would get a bitrate sightly inferior to the 130kbps target... (i remember about 115 kbps/s for a CDA but it was "chanson" not classical thought)
Anyway not a matter but i thought comparing the two codec with VBR would have been helpfull...
smile.gif
Go to the top of the page
+Quote Post
Gabriel
post Nov 15 2004, 22:54
Post #9


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



Preset medium (similar to V4 in 3.96) is targetting 150-160kbps overall
Go to the top of the page
+Quote Post
Tang
post Nov 15 2004, 23:00
Post #10





Group: Members (Donating)
Posts: 158
Joined: 27-January 04
Member No.: 11536



QUOTE (Gabriel @ Nov 15 2004, 01:54 PM)
Preset medium (similar to V4 in 3.96) is targetting 150-160kbps overall
*

Hi Gabriel,
Thanks for the info... Seems that i've been mistaken cause i used -apm only once time with one CDA of song (live of Jeanne Cherhal) which gave me an average bitrate about 115kbps...
Anyway a single shot encoding (~tracks thought) didi'nt allow généralisation, that's why i mistaken...
Sorry.. smile.gif

regards,
Tanguy
Go to the top of the page
+Quote Post
[proxima]
post Nov 16 2004, 00:26
Post #11





Group: Members
Posts: 197
Joined: 12-October 02
From: Italy
Member No.: 3537



QUOTE (guruboolez @ Nov 15 2004, 02:06 PM)
Are you talking about one precise sample

Those coloured orange in your test.
I ask because i have found some samples (LisztBMinor, Blackwater) where VBR 3.96 still suffers a little with ringing/HF problems even with --athaa-sensitivity 1.


--------------------
WavPack 4.3 -mfx5
LAME 3.97 -V5 --vbr-new --athaa-sensitivity 1
Go to the top of the page
+Quote Post
Lyx
post Nov 16 2004, 04:16
Post #12





Group: Members
Posts: 3353
Joined: 6-July 03
From: Sachsen (DE)
Member No.: 7609



Just for understanding:

3.90.3 ABR and 3.96.1 VBR both average at 130kbit
Yet still in your test the bitrates on the samples are usually higher with 3.96.1 VBR

Am i right when concluding that 3.96.1 VBR appears to identify "problem-samples" more reliably and ups the bitrate? Or am i misinterpreting something?

- Lyx


--------------------
I am arrogant and I can afford it because I deliver.
Go to the top of the page
+Quote Post
guruboolez
post Nov 16 2004, 09:47
Post #13





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE ([proxima] @ Nov 16 2004, 12:26 AM)
QUOTE (guruboolez @ Nov 15 2004, 02:06 PM)
Are you talking about one precise sample

Those coloured orange in your test.
I ask because i have found some samples (LisztBMinor, Blackwater) where VBR 3.96 still suffers a little with ringing/HF problems even with --athaa-sensitivity 1.
*

I can't remember exactly what problems I heard on these samples (I'm at work, and I don't have access to the log), but 3.96.1 problems are not limited to ringing. Correcting this problem won't solve all issues.
Go to the top of the page
+Quote Post
dev0
post Nov 16 2004, 10:08
Post #14





Group: Developer
Posts: 1679
Joined: 23-December 01
From: Germany
Member No.: 731



QUOTE (Lyx @ Nov 16 2004, 04:16 AM)
Am i right when concluding that 3.96.1 VBR appears to identify "problem-samples" more reliably and ups the bitrate? Or am i misinterpreting something?
*

Yes, that's what makes VBR so wonderful.
But it also makes the target bitrate less controllable.
But across a wide range of music/genres it should average around 130kbps.


--------------------
"To understand me, you'll have to swallow a world." Or maybe your words.
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 21st December 2014 - 06:28