IPB

Welcome Guest ( Log In | Register )

> Upload forum rules

- No over 30 sec clips of copyrighted music. Cite properly and never more than necessary for the discussion.


- No copyrighted software without permission.


- Click here for complete Hydrogenaudio Terms of Service

3 Pages V   1 2 3 >  
Closed TopicStart new topic
LAME 3.96b regression examples, Post any possible regressions here!
Jebus
post Mar 8 2004, 01:55
Post #1





Group: Developer
Posts: 1320
Joined: 17-March 03
From: Calgary, AB
Member No.: 5541



This is NOT a general 3.96b ABX results thread. This is just for people to post (any) samples which regress from 3.90.3. This means that I only want cases where 3.96 is WORSE than 3.90.3.

Please try to keep this on-topic. The idea is that if this thread stays empty, we should push for 3.96 to become official. And if it doesn't stay empty, samples will help the LAME devs with future versions.

Any and all settings are fine... just make sure you post how you encoded it, and link to samples.

This post has been edited by Jebus: Mar 9 2004, 10:47
Go to the top of the page
+Quote Post
Moitah
post Mar 8 2004, 02:06
Post #2





Group: Members
Posts: 193
Joined: 5-June 02
From: Virginia Beach, VA
Member No.: 2227



Here's a sample which I can ABX (15/16) when encoded with 3.96 --preset standard: Dream Theater - The Dance of Eternity (8 sec, FLAC). Listen to the part where the snair hits 4 times, about 1.5 seconds in. I first noticed this while testing --preset 128, it happened in 3.96 but I didn't notice the same artifact in 3.90.3 --alt-preset 128.
Go to the top of the page
+Quote Post
PVNC
post Mar 8 2004, 02:14
Post #3





Group: Members
Posts: 16
Joined: 8-February 04
From: Ottawa, Canada
Member No.: 11844



I tried a 28-second sample from Tool's track (-) Ions. I chose this to test first, as it gets an average bitrate of 384 kbps using Vorbis GT3b1 at q5.

Original sample is in FLAC format.

I encoded mp3 versions using only the commandline --alt-preset 128 in RazorLame. I used LAME 3.90.3 modified and 3.96b1.

FLAC and mp3 versions have ReplayGain information applied.

I found that the 3.96b1 version has some kind of distortion during the lower volume parts of this clip - between the peak volume levels of the buzzing. I was able to use this to ABX 3.96 from 3.90.3 8/8.

(-) Ions
Go to the top of the page
+Quote Post
guruboolez
post Mar 8 2004, 02:19
Post #4





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



Isn't the upload forum a better place for this topic? Maybe should someone create a topic dedicated to this collection, or move this one.
Go to the top of the page
+Quote Post
Dologan
post Mar 8 2004, 02:25
Post #5





Group: Members (Donating)
Posts: 478
Joined: 22-November 01
From: United Kingdom
Member No.: 519



Also, I would encourage people to also post improvements from the old encoder, since otherwise we might be skewing our preception. Usually, improvements also introduce other minor flaws that nonetheless end up making the thing a good trade-off.
Go to the top of the page
+Quote Post
PVNC
post Mar 8 2004, 02:33
Post #6





Group: Members
Posts: 16
Joined: 8-February 04
From: Ottawa, Canada
Member No.: 11844



QUOTE (guruboolez @ Mar 7 2004, 05:19 PM )
Isn't the upload forum a better place for this topic? Maybe should someone create a topic dedicated to this collection, or move this one.


I agree. I didn't know that my uploaded sample would end up on the portal. It was rather embarrassing.
Go to the top of the page
+Quote Post
westgroveg
post Mar 8 2004, 02:49
Post #7





Group: Members
Posts: 1236
Joined: 5-October 01
Member No.: 220



CODE
ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\ringing-edit--APS-3.90.3.wav
   9 out of 10, pval = 0.011


CODE
ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\ringing-edit--APS-3.96b1.wav
  9 out of 10, pval = 0.011


CODE
ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\ringing-edit--APS-3.96b1.wav
   13 out of 16, pval = 0.011


There is a HF hiss present (0:01-0:02).

This post has been edited by westgroveg: Mar 8 2004, 09:29
Attached File(s)
Attached File  ringing_edit.wav ( 1.08MB ) Number of downloads: 311
 
Go to the top of the page
+Quote Post
westgroveg
post Mar 8 2004, 02:52
Post #8





Group: Members
Posts: 1236
Joined: 5-October 01
Member No.: 220



CODE
ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\castanets-edit-2-APS-3.96b1.wav
   12 out of 17, pval = 0.072


Guitar sounds watery.
Attached File(s)
Attached File  castanets_edit_2.wav ( 172.13K ) Number of downloads: 330
 
Go to the top of the page
+Quote Post
westgroveg
post Mar 8 2004, 03:08
Post #9





Group: Members
Posts: 1236
Joined: 5-October 01
Member No.: 220



Moitah's The Dance of Eternity sample uploaded at HA,
Attached File(s)
Attached File  doesnair.flac ( 872.58K ) Number of downloads: 271
 
Go to the top of the page
+Quote Post
Moitah
post Mar 8 2004, 19:35
Post #10





Group: Members
Posts: 193
Joined: 5-June 02
From: Virginia Beach, VA
Member No.: 2227



Play - Is It Love (4 sec, FLAC) (try this link first, to save HA bandwidth)

--preset standard: This takes a lot of concentration for me to ABX, I got 22/32, 23/32, and 14/16 which adds up to 59/80 (p<0.001). The first time the high note is played on the guitar (1.2 secs in) sounds smoother in the original.

This post has been edited by Moitah: Mar 8 2004, 19:40
Attached File(s)
Attached File  isitloveintro.flac ( 447.13K ) Number of downloads: 247
 
Go to the top of the page
+Quote Post
amano
post Mar 8 2004, 21:17
Post #11





Group: Members
Posts: 483
Joined: 1-December 02
Member No.: 3949



QUOTE (westgroveg @ Mar 7 2004, 05:52 PM)
CODE
ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\castanets-edit-2-APS-3.96b1.wav
   12 out of 17, pval = 0.072


Guitar sounds watery.

As this is a regression thread, it dosn't make sense to just post just non-transparent samples. You have to compare them to the ABX results of 3.90.3.

And please don't post ABX results that aren't statistically valid. 12 out of 17 is certainly not a statistically convincing result. Otherwise this thread will be spammed with useless information.

This post has been edited by amano: Mar 8 2004, 21:37
Go to the top of the page
+Quote Post
amano
post Mar 8 2004, 21:23
Post #12





Group: Members
Posts: 483
Joined: 1-December 02
Member No.: 3949



QUOTE (Moitah @ Mar 8 2004, 10:35 AM)
Play - Is It Love (4 sec, FLAC) (try this link first, to save HA bandwidth)

--preset standard: This takes a lot of concentration for me to ABX, I got 22/32, 23/32, and 14/16 which adds up to 59/80 (p<0.001).  The first time the high note is played on the guitar (1.2 secs in) sounds smoother in the original.

I don't think that is valid to add up session results (maybe some other more in depth in the ABX science may tell us for sure).

And please compare your results with 3.90.3. To be fair chose the same session procedure. and try to ABX 3.90.3 against 3.96.

EDIT: I think everyone should act according to tigre's proposals in the LAME poll thread:

QUOTE
# Posting results in the thread requires:

    * Upload or link to sample
    * ABX results Original<->3.90.3, Original<->3.96, 3.90.3<->3.96, with detailed description of the difference(s)
    * Report about software/hardware used: Soundcard (resampling?), Player/ABXtool, DSPs (shouldn't be allowed, besides resampling to 48kHz and volume reduction/replaygain to prevent clipping <- both a 'must'), Amplifier, Speakers/Headphones

# Results must be confirmed by someone else before they are included in 'official' statistic, p-values must be < 0.05 for at least 2 people.


This post has been edited by amano: Mar 8 2004, 21:35
Go to the top of the page
+Quote Post
indybrett
post Mar 8 2004, 21:27
Post #13





Group: Members (Donating)
Posts: 1350
Joined: 4-March 02
From: Indianapolis, IN
Member No.: 1440



Since we have a regression thread, maybe we should also have a progression thread.


--------------------
flac>fb2k>kernel streaming>audiophile 2496>magni>dt990 pro
Go to the top of the page
+Quote Post
westgroveg
post Mar 9 2004, 06:27
Post #14





Group: Members
Posts: 1236
Joined: 5-October 01
Member No.: 220



QUOTE
As this is a regression thread, it dosn't make sense to just post just non-transparent samples. You have to compare them to the ABX results of 3.90.3.

And please don't post ABX results that aren't statistically valid. 12 out of 17 is certainly not a statistically convincing result. Otherwise this thread will be spammed with useless information.

It was the only test I did with 3.96, I also did 1 ABX test with 3.90.3 got 5/5 thus my post of the sample.

Anyway your not a moderator, you didn't start the thread or even post a sample so I don't really see what your interest is, you just must like telling everyone what to do wink.gif .

(HA Wiki)
QUOTE
A difference is concluded to be heard when 13 correct identifications out of 16 trials is achieved

This is what we are expected to score for a valid problem sample, people, not what amano wants us to do.

This post has been edited by westgroveg: Mar 9 2004, 08:13
Go to the top of the page
+Quote Post
Jebus
post Mar 9 2004, 06:42
Post #15





Group: Developer
Posts: 1320
Joined: 17-March 03
From: Calgary, AB
Member No.: 5541



Westgroveg, i appreciate the results you are posting but PLEASE, i specifically requested that ONLY regressions from 3.90.3 be posted here, NOT general ABX results.

EDIT: Corrected Westgroveg's name

This post has been edited by Jebus: Mar 9 2004, 09:45
Go to the top of the page
+Quote Post
westgroveg
post Mar 9 2004, 06:55
Post #16





Group: Members
Posts: 1236
Joined: 5-October 01
Member No.: 220



QUOTE (Jebus @ Mar 9 2004, 05:42 PM)
Westgroveq, i appreciate the results you are posting but PLEASE, i specifically requested that ONLY regressions from 3.90.3 be posted here, NOT general ABX results.

My first sample as you can see I was able to ABX with both 3.90.3 & 3.96

My second admittedly is questionable but I also counter checked with 3.90.3, what's the problem?

& my nck is westgroveg.
Go to the top of the page
+Quote Post
freakngoat
post Mar 9 2004, 09:01
Post #17





Group: Members
Posts: 170
Joined: 10-January 02
From: Manhattan Beach
Member No.: 979



QUOTE (westgroveg @ Mar 9 2004, 05:55 AM)
My first sample as you can see I was able to ABX with both 3.90.3 & 3.96

My second admittedly is questionable but I also counter checked with 3.90.3, what's the problem?

Sorry, don't mean to butt in, but they mean your ABX results aren't worse with 3.96 than they were with 3.90.3; they appear to be equal or even better. Regression testing means running past tests over again on a new version of software to make sure that nothing broke or got worse (regressed) with the newer version.
Go to the top of the page
+Quote Post
Jebus
post Mar 9 2004, 09:47
Post #18





Group: Developer
Posts: 1320
Joined: 17-March 03
From: Calgary, AB
Member No.: 5541



QUOTE (westgroveg @ Mar 8 2004, 09:55 PM)
QUOTE (Jebus @ Mar 9 2004, 05:42 PM)
Westgroveq, i appreciate the results you are posting but PLEASE, i specifically requested that ONLY regressions from 3.90.3 be posted here, NOT general ABX results.

My first sample as you can see I was able to ABX with both 3.90.3 & 3.96

My second admittedly is questionable but I also counter checked with 3.90.3, what's the problem?

& my nck is westgroveg.

As Freakngoat just posted, the idea is that we ONLY post cases where 3.96 is WORSE than 3.90.3. You just posted cases where they both mess up - which isn't the point of this thread.
Go to the top of the page
+Quote Post
westgroveg
post Mar 9 2004, 10:43
Post #19





Group: Members
Posts: 1236
Joined: 5-October 01
Member No.: 220



QUOTE (freakngoat @ Mar 9 2004, 08:01 PM)
QUOTE (westgroveg @ Mar 9 2004, 05:55 AM)
My first sample as you can see I was able to ABX with both 3.90.3 & 3.96

My second admittedly is questionable but I also counter checked with 3.90.3, what's the problem?

Sorry, don't mean to butt in, but they mean your ABX results aren't worse with 3.96 than they were with 3.90.3; they appear to be equal or even better. Regression testing means running past tests over again on a new version of software to make sure that nothing broke or got worse (regressed) with the newer version.

Oh, sorry I misunderstood, I thought the thread asked, samples which are problematic for both 3.90.3 & 3.96, sorry Jebus.

rolleyes.gif
Go to the top of the page
+Quote Post
Jebus
post Mar 9 2004, 10:47
Post #20





Group: Developer
Posts: 1320
Joined: 17-March 03
From: Calgary, AB
Member No.: 5541



np smile.gif

I added a clarification sentence to the first post
Go to the top of the page
+Quote Post
westgroveg
post Mar 9 2004, 11:44
Post #21





Group: Members
Posts: 1236
Joined: 5-October 01
Member No.: 220



I just had a go at PVNC's (-) Ions sample (I used the contained mp3's for ABX)
CODE
ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\3.96b1 (-) Ions.wav
   16 out of 16, pval < 0.001
C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\3.96b1 (-) Ions.wav vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\3.90.3 (-) Ions.wav
   10 out of 10, pval < 0.001


Both have a huge distortion (0:02-->0:04), 3.90.3 sounds slightly better.

This post has been edited by westgroveg: Mar 9 2004, 11:54
Go to the top of the page
+Quote Post
westgroveg
post Mar 9 2004, 12:06
Post #22





Group: Members
Posts: 1236
Joined: 5-October 01
Member No.: 220



QUOTE (Moitah @ Mar 8 2004, 01:06 PM)
Here's a sample which I can ABX (15/16) when encoded with 3.96 --preset standard: Dream Theater - The Dance of Eternity (8 sec, FLAC).  Listen to the part where the snair hits 4 times, about 1.5 seconds in.  I first noticed this while testing --preset 128, it happened in 3.96 but I didn't notice the same artifact in 3.90.3 --alt-preset 128.

I find these rock samples hard to ABX, can you tell us what type of artifact to listen for?
Go to the top of the page
+Quote Post
Moitah
post Mar 9 2004, 15:56
Post #23





Group: Members
Posts: 193
Joined: 5-June 02
From: Virginia Beach, VA
Member No.: 2227



QUOTE (westgroveg @ Mar 9 2004, 06:06 AM)
I find these rock samples hard to ABX, can you tell us what type of artifact to listen for?

I'm not sure what it's called, pre-echo maybe. If you start with 128 it should be easier.

I need to do more testing with my second sample, it might be happening in 3.90.3 as well.
Go to the top of the page
+Quote Post
2Bdecided
post Mar 9 2004, 16:39
Post #24


ReplayGain developer


Group: Developer
Posts: 5258
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



QUOTE (amano @ Mar 8 2004, 08:23 PM)
I don't think that is valid to add up session results (maybe some other more in depth in the ABX science may tell us for sure).

You should always add all ABX result together for the same sample. It is statistically valid.

What you should never do is do more than one set of ABX tests, but only report some of them - you always need to report all the ABX testing you did with a sample. Selective ABX results are not valid.


Also, watching the result, and stopping when it's "statistically valid" isn't right either, but I haven't seen an explanation of what the correct statistics are in this case. The advice has always been to decide how many trials you will take before you start. But if you watch your result, then the "standard" statistics (which are found in most ABX programs) are wrong.

Cheers,
David.
Go to the top of the page
+Quote Post
Moitah
post Mar 9 2004, 17:50
Post #25





Group: Members
Posts: 193
Joined: 5-June 02
From: Virginia Beach, VA
Member No.: 2227



My second sample isn't a regression, I ABXed 16/20 and 12/12 (total 28/32, pval < 0.001) for 3.90.3 --alt-preset standard (I think I was able to ABX 3.90.3 better because I knew what to listen for, not because it sounded worse than 3.96).
Go to the top of the page
+Quote Post

3 Pages V   1 2 3 >
Closed TopicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 22nd November 2014 - 16:57