IPB

Welcome Guest ( Log In | Register )

2 Pages V   1 2 >  
Reply to this topicStart new topic
Why does LAME 3.93 have higher SNR than 3.98?, was: "Why is ... much better than ..."
Istrebitel
post Sep 28 2011, 13:22
Post #1





Group: Members
Posts: 54
Joined: 20-October 10
Member No.: 84756



Greetings.

I am analysing SNR (signal to noise ratio) of files and i noticed one exact problem: On CBR or ABR, same bitrate on newer 3.98.4 would produce (sometimes much) worse results than old 3.93.1
For example, on 192 kbits on my test sample (Vivaldi 4 seasons, CD rip 16bit 44100hz) endoding would produce ~27 on newer lame versus ~34 on older lame!

Is this a well known bug in 3.98.4? Or why does a latter version produce worser signal-to-noise ration than earlier version?

This post has been edited by Istrebitel: Sep 28 2011, 13:23
Go to the top of the page
+Quote Post
pdq
post Sep 28 2011, 15:57
Post #2





Group: Members
Posts: 3402
Joined: 1-September 05
From: SE Pennsylvania
Member No.: 24233



First tell us how you are measuring SNR, and then maybe we can shed some light on this.
Go to the top of the page
+Quote Post
saratoga
post Sep 28 2011, 16:36
Post #3





Group: Members
Posts: 4957
Joined: 2-September 02
Member No.: 3264



QUOTE (Istrebitel @ Sep 28 2011, 08:22) *
Or why does a latter version produce worser signal-to-noise ration than earlier version?


Probably due to differences in the low pass filter settings. Assuming you have the same low pass and bitrate settings, the SNR doesn't really depend much on the encoder.

QUOTE (Istrebitel @ Sep 28 2011, 08:22) *
Is this a well known bug in 3.98.4?


No. SNR is not meaningful for mp3 files, so theres no expectation that higher numbers are better, or that it should have any particular value. Its basically just a random number determined by how the mp3 format works, the lowpass/sample rate settings, and the bitrate.
Go to the top of the page
+Quote Post
Istrebitel
post Sep 29 2011, 08:13
Post #4





Group: Members
Posts: 54
Joined: 20-October 10
Member No.: 84756



2pdq

I am measuring SNR.... erm... like it is supposed to be measured.

First i offset the resulting file to match the source PCM file, measure to measure.
Then i calculate Asignal**2 / Anoise**2, as said in wikipedia http://en.wikipedia.org/wiki/Signal-to-noise_ratio

This means that, for example, for each second (44100 measures) (assuming mono signal for simplicity) i am calculating (f1[0]**2 + f1[1]**2 + ... + f1[44099]**2) / ((f1[0] - f2[0])**2 + (f1[1] - f2[1])**2 + ... + (f1[44099] - f2[44099])**2), where f1[i] is source signal and f2[i] is encoded-decoded signal, and that is the SNR for this second of the signal.

So, medium value over the file is ~27 when comparing source file to 3.98.4 encoded file, and ~34 when comparing source file to 3.93.1 encoded file.
I can upload the files in question so you could check in your own SNR measuring program if you wish.

2 saratoga

I see, how do i get to know what ow pass settings i am using? Bitrate is the same, --cbr -b 192
I know SNR isnt totally meaningful to understand mp3 file quality, but still, it does corellate with quality. SNR for bigger bitrate would always be bigger than for lower bitrate.
Such a difference in signal-to-noise ration feels like the quality of 3.98.4 went down compared to 3.93.1.
Here is an example of topic where person says that listening tests confirm that http://www.claessonedwards.com/forum/viewtopic.php?p=9839

This post has been edited by Istrebitel: Sep 29 2011, 09:08
Go to the top of the page
+Quote Post
greynol
post Sep 29 2011, 08:21
Post #5





Group: Super Moderator
Posts: 10000
Joined: 1-April 04
From: San Francisco
Member No.: 13167



Not "isn't totally meaningful," rather such measurements are useless (except for extreme/trivial/uninteresting cases), and are forbidden as a means to support claims about the sound quality of lossy compression on this forum.

These are the terms to which you agreed in order to participate here (specifically #8 in this case):
http://www.hydrogenaudio.org/forums/index.php?showtopic=3974

If you aren't already familiar, you should familiarize yourself with what are the proper means to support sound quality claims:
http://www.hydrogenaudio.org/forums/index....showtopic=16295

This post has been edited by greynol: Sep 29 2011, 08:29


--------------------
I should publish a list of forum idiots.
Go to the top of the page
+Quote Post
Kohlrabi
post Sep 29 2011, 08:43
Post #6





Group: Super Moderator
Posts: 1046
Joined: 12-March 05
From: Kiel, Germany
Member No.: 20561



I once did some tests evaluating the PSNR metric for different encoders. As expected, the PSNR didn't have a high correlation with perceived audio quality. I compared AoTuV vorbis, LAME and Nero AAC. Nero AAC produced much higher PSNR at most bitrates than LAME, and AoTuV was mostly below those two. But listening tests so far have shown that AoTuV is at least tied to LAME and Nero AAC in terms of perceived quality if not better. Also, following the video encoding world where the metric PSNR is often used, one can see that it also has only some correlation with perceived quality. Encoders with good psychoacoustic/psychovisual models do not necessarily produce higher PSNR numbers.

In the video world is has become a bit common now to use the SSIM metric to evaluate the visual quality, from what I read it correlates better with the perceived quality than (P)SNR. You can find some papers about possible audio quality metrics, including one about SSIM, on Google Scholar, maybe it includes some useful ones. I would be quite interested to see tests of correlation of results of public listening tests with said metrics.

That said, the agreed upon method on Hydrogenaudio to evaluate lossy encoders is the blind listening test. It basically says that the only meaningful metric to assess a lossy encoder is the perceived difference from the source material, and the listener is not allowed to know which sample is which (blind test) during evaluation. If you can not reliably tell the lossy encode from the source, the lossy encode is considered transparent, i.e. perceptually indistinguishable from the source. That means that the encoder works properly and well at that chosen settings. So a more meaningful test than comparing SNR values would be ABXing LAME 3.93.1 and LAME 3.98.4 encodes. I expect them to be quite close contestors.

This post has been edited by Kohlrabi: Sep 29 2011, 09:04


--------------------
Ceterum censeo Masterdiskem esse delendam.
Go to the top of the page
+Quote Post
Istrebitel
post Sep 29 2011, 09:37
Post #7





Group: Members
Posts: 54
Joined: 20-October 10
Member No.: 84756



2 greynol
I dont make any claims. Really. I am not making a statement. If it did sound like that please forgive me, english isnt my native tongue.
All i wonder is why. Why does it happen. Why does latter lame provide more noisy signal (less SNR). I'm not saying "why does latter lame have lower quality" however even that wouldnt be a statement, more of a wrong formed question (a question that implies something that isnt a fact, like a question "Does your mother know you are a gay" impiles you are a gay, and whichever you answer, yes or no, makes you agree with it).

Once again, if i sound like i make a statement please forgive me. I do not. I am just showing SNR calculations i made and ask why can it happen like that.

2 Kohlrabi
Yeah, i do understand listening tests are the way to determine quality. However, i am not able to do that.
So, i am right now stuck with SNR. And i have now tested 3.90.3 as well - both 3.90.31 and 3.93.1 provide a bigger SNR than 3.98.3

Now my question is why can this happen. Did defaults change? Did pshycoacoustic model change? Did something else change?
In other words, my queston is "Can i do something to my 3.98.3 to make it produce a file of similar SNR to 3.93.1 or 3.90.3?"
Go to the top of the page
+Quote Post
Istrebitel
post Sep 29 2011, 09:42
Post #8





Group: Members
Posts: 54
Joined: 20-October 10
Member No.: 84756



Here is some output data:
(--resample 44100 is effectively not used since input file is 44100)

lame_3_93_1.exe --cbr -b 192 ..\CNT\8.wav --resample 44100 -m s tmp
LAME version 3.93 MMX (http://www.mp3dev.org)
(Win32 binaries from: http://www.hot.ee/smpman/mp3)
CPU features: i387, MMX (ASM used), SIMD, SIMD2
Using polyphase lowpass filter, transition band: 18671 Hz - 19205 Hz
Encoding ..\CNT\8.wav to tmp
Encoding as 44.1 kHz 192 kbps stereo MPEG-1 Layer III (7.3x) qval=2
Frame | CPU time/estim | REAL time/estim | play/CPU | ETA
7722/7724 (100%)| 0:13/ 0:13| 0:13/ 0:13| 15.499x| 0:00
average: 192.0 kbps LR: 7725 (100.0%)

lame_3_90_3.exe --cbr -b 192 ..\CNT\8.wav --resample 44100 -m s tmp
..\lame_3_90_3.exe: unrec option --cbr
LAME version 3.90.3 MMX (http://www.mp3dev.org/)
CPU features: i387, MMX (ASM used), SIMD, SIMD2
Using polyphase lowpass filter, transition band: 18671 Hz - 19205 Hz
Encoding ..\CNT\8.wav to tmp
Encoding as 44.1 kHz 192 kbps stereo MPEG-1 Layer III (7.3x) qval=5
Frame | CPU time/estim | REAL time/estim | play/CPU | ETA
7722/7725 (100%)| 0:16/ 0:16| 0:16/ 0:16| 12.534x| 0:00
average: 192.0 kbps LR: 7725 (100.0%)

lame_3_98_4.exe --cbr -b 192 ..\CNT\8.wav --resample 44100 -m s tmp
LAME 3.98.4 32bits (http://lame.sf.net/)
CPU features: MMX (ASM used), SSE (ASM used), SSE2
Using polyphase lowpass filter, transition band: 18671 Hz - 19205 Hz
Encoding ..\CNT\8.wav to tmp
Encoding as 44.1 kHz stereo MPEG-1 Layer III (7.3x) 192 kbps qval=3
Frame | CPU time/estim | REAL time/estim | play/CPU | ETA
7724/7724 (100%)| 0:22/ 0:22| 0:22/ 0:22| 8.9612x| 0:00
-------------------------------------------------------------------------------
kbps LR % long switch short %
192.0 100.0 99.8 0.1 0.1
Writing LAME Tag...done
ReplayGain: -2.3dB

This post has been edited by Istrebitel: Sep 29 2011, 09:51
Go to the top of the page
+Quote Post
robert
post Sep 29 2011, 10:22
Post #9


LAME developer


Group: Developer
Posts: 788
Joined: 22-September 01
Member No.: 5



Since LAME version 3.94, the ABR/CBR/VBR modes use some presets, so no more plain old GPsycho psy model. (look out for NSPsycho, Naoki Shibata's psy tunings)

http://lame.cvs.sourceforge.net/viewvc/lam...l?revision=HEAD

QUOTE
Gabriel Bouvigne:
    work on presets
  • use presets by default for cbr/abr
  • use presets by default for vbr


This post has been edited by robert: Sep 29 2011, 10:22
Go to the top of the page
+Quote Post
Istrebitel
post Sep 29 2011, 11:23
Post #10





Group: Members
Posts: 54
Joined: 20-October 10
Member No.: 84756



So, robert, can i using some input parameters "emulate" 3.93 behavior on 3.98? I mean, can i tweak 3.98 by changing some settings so it would encode like 3.93 does by default?
Go to the top of the page
+Quote Post
shadowking
post Sep 29 2011, 11:48
Post #11





Group: Members
Posts: 1526
Joined: 31-January 04
Member No.: 11664



Before v3.94 you would need to use --alt-preset cbr 192

That is CBR using presets like 3.98.4. --cbr -b is the old code until 3.94. Everything after that default to presets - -b192, --cbr 192, --alt-preset 192 are all the same since 3.94



--------------------
Wavpack -b450s0.7
Go to the top of the page
+Quote Post
robert
post Sep 29 2011, 11:56
Post #12


LAME developer


Group: Developer
Posts: 788
Joined: 22-September 01
Member No.: 5



QUOTE (Istrebitel @ Sep 29 2011, 12:23) *
So, robert, can i using some input parameters "emulate" 3.93 behavior on 3.98? I mean, can i tweak 3.98 by changing some settings so it would encode like 3.93 does by default?


No.
Go to the top of the page
+Quote Post
dhromed
post Sep 29 2011, 12:30
Post #13





Group: Members
Posts: 1314
Joined: 16-February 08
From: NL
Member No.: 51347



QUOTE (Istrebitel @ Sep 29 2011, 10:37) *
Now my question is why can this happen. Did defaults change? Did pshycoacoustic model change? Did something else change?
In other words, my queston is "Can i do something to my 3.98.3 to make it produce a file of similar SNR to 3.93.1 or 3.90.3?"


That's not "in other words"; those are two different questions.

Question 1. What changed?
Answer: maybe the LAME changelog has more details, or a LAME dev might respond to email in a timely manner. The answer may or may not be useful to you.

Question 2. Can I set some settings to get back the old SNR?
Answer: why do you want to do this? Purely for academic purposes? To learn a little more about how LAME works?

This post has been edited by dhromed: Sep 29 2011, 12:33
Go to the top of the page
+Quote Post
Istrebitel
post Sep 29 2011, 13:33
Post #14





Group: Members
Posts: 54
Joined: 20-October 10
Member No.: 84756



Okay, i understand my questions are maybe too uncertain, then let me rephrase it:

Are you sure that i am not doing something wrong, which makes me get less SNR on 3.98 than on 3.93?
And - are you sure that the reduction in SNR i am getting when encoding with 3.98 when compared to 3.93 does not at all indicate a drop in quality?
And - what would be your most probable explanation of why did such SNR drop happen when going from 3.93 to 3.98?

PS: Yes, you can call this interest of mine "purely academic".
Go to the top of the page
+Quote Post
pdq
post Sep 29 2011, 14:30
Post #15





Group: Members
Posts: 3402
Joined: 1-September 05
From: SE Pennsylvania
Member No.: 24233



Please stop referring to this as SNR. What you are looking at is the remainder after subtracting the decoded file from the original, which, as has already been pointed out, typically bears little or no relationship with quality.

Several times per month someone arrives here with the revelation that all you have to do is subtract the original from the decoded file and miraculously you now have a numeric representation of quality. It doesn't work. So please stop pestering us with this nonsense and come back when you have some serious questions that you want to ask.
Go to the top of the page
+Quote Post
greynol
post Sep 29 2011, 16:57
Post #16





Group: Super Moderator
Posts: 10000
Joined: 1-April 04
From: San Francisco
Member No.: 13167



QUOTE (Istrebitel @ Sep 29 2011, 01:37) *
a wrong formed question

This is exactly the underlying problem with this topic of yours.

QUOTE (Istrebitel @ Sep 29 2011, 01:37) *
I am just showing SNR calculations i made and ask why can it happen like that.

Welcome to the wonderful world of psychoacoustics!

QUOTE (Istrebitel @ Sep 29 2011, 01:37) *
both 3.90.31 and 3.93.1 provide a bigger SNR than 3.98.3

...and completely irrelevant if you're trying to determine whether there is a difference in perceived sound quality.

QUOTE (Istrebitel @ Sep 29 2011, 01:37) *
Can i do something to my 3.98.3 to make it produce a file of similar SNR to 3.93.1 or 3.90.3?

Which begs the question: even if you could, why would you want to?


--------------------
I should publish a list of forum idiots.
Go to the top of the page
+Quote Post
saratoga
post Sep 29 2011, 17:22
Post #17





Group: Members
Posts: 4957
Joined: 2-September 02
Member No.: 3264



QUOTE (Istrebitel @ Sep 29 2011, 08:33) *
And - are you sure that the reduction in SNR i am getting when encoding with 3.98 when compared to 3.93 does not at all indicate a drop in quality?


Perhaps I was unclear when I said that your SNR measurement was a meaningless number with no connection to quality. Let me be more clear: its completely worthless. No use at all. You might as well flip coins.

QUOTE (Istrebitel @ Sep 29 2011, 08:33) *
And - what would be your most probable explanation of why did such SNR drop happen when going from 3.93 to 3.98?


Have you verified that the low pass filter settings are the same?
Go to the top of the page
+Quote Post
Istrebitel
post Sep 30 2011, 08:02
Post #18





Group: Members
Posts: 54
Joined: 20-October 10
Member No.: 84756



On topic that what i am calculating is not SNR:
Is wikipedia wrong then? I am calculating exactly what SNR it is said to be on wikipedia, Psignal/Pnoise.
http://en.wikipedia.org/wiki/Signal-to-noise_ratio

On topic that SNR is worthless and does not relate to quality:
How do you explain then that same file encoded with lesser bitrate AWAYS has smaller SNR than same file encoded with bigger bitrate.
And (i hope) there is no argument that SNR is telling EXACTLY how much the signal was distorted. Since it is by definition SIGNAL to NOISE ratio. The bigger it is, the less noise was added. The lower it is, the more noise was added.
However, there is also no argument that signal distortion != percieved distortion, since our ear has its own hearing quirks. Thats what psychoacoustics are about. And I never stated that SNR corelates to HEARING QUALITY and i know SNR does not measure that.
But I am talking about signal distortion here. And it is clear that 3.98.3 provides more signal distortion than 3.93.1. Now the argument that this distortion does not mean by itself that it sounds worse is not relevant to me. I never argued about that! What i need to know is wether i am doing something wrong with 3.98.3 that i receive a more distorted file, or wether i am doing nothing wrong, and drop in SNR is intended, and indeed, less SNR brings more HEARING QUALITY in this case.
Just look here http://www.claessonedwards.com/forum/viewtopic.php?p=9839
Believe your eyes. More waveform distortion with 3.98.3, less with 3.93.1

About lowpass:
I typed the console output. Since all three say "Using polyphase lowpass filter, transition band: 18671 Hz - 19205 Hz" can i assume that lowpass settings are the same? What parameters shall i input to make sure that ALL filter settings are the same?

About why do i want it:
If you know how to do it, tell me. If you dont, tell me you dont know how to do it. If i cannot explain to you importance of something it doesnt mean it is not important to me. A kid cannot explain importance of healthy food, or how does it affect his body health, but it is still important to him, right?

This post has been edited by Istrebitel: Sep 30 2011, 08:15
Go to the top of the page
+Quote Post
alexeysp
post Sep 30 2011, 09:42
Post #19





Group: Members
Posts: 130
Joined: 3-April 09
Member No.: 68627



QUOTE (Istrebitel @ Sep 30 2011, 10:02) *
On topic that SNR is worthless and does not relate to quality:
How do you explain then that same file encoded with lesser bitrate AWAYS has smaller SNR than same file encoded with bigger bitrate.


Obviously, lower bitrate means higher level of distortion (apart from, maybe, specific artificial cases). But lower bitrate does not universally imply lower quality, depending on particular definition of the latter. So please, either stop using the word "quality", or provide us with your definition of it before talking about lower/higher quality.

QUOTE
What i need to know is wether i am doing something wrong with 3.98.3 that i receive a more distorted file, or wether i am doing nothing wrong, and drop in SNR is intended, and indeed, less SNR brings more HEARING QUALITY in this case.


How can you say you do something wrong if you use exactly same command line parameters? Obviously, something within the encoder itself has changed, probably the masking thresholds used for perceptual analysis. You need to compare the source code for different versions to find out the exact reasons.

As for whether the decreased SNR means degradation in codec quality - no, there is no direct connection, since preserving the waveform (i.e. minimizing the absolute amount of distortion) is not the codec's purpose, nor a design goal. If you want some numeric characteristic of the codec quality then first, you need to compute SNR in frequency domain, and second, the computation must be weighted according to masking thresholds calculated for each analysis window. And even then you can't say that the codec with higher metric is "better", since both may appear transparent in a listening test.

If you are trying to enforce codec to serve the different purpose, then you need to develop the different analysis model. If your goal is to maximize average SNR, then you need to replace the psychoacoustic model with the model that would calculate scalefactors according to the maximum possible average SNR criterion (this probably would mean uniform quantization noise ditribution across all bands, although I'm not sure). Of course you should not hope for perceptual transparency at any bitrate in this case.

Go to the top of the page
+Quote Post
Istrebitel
post Sep 30 2011, 10:33
Post #20





Group: Members
Posts: 54
Joined: 20-October 10
Member No.: 84756



2 alexeysp
Okay, i will stop using the world quality in the sense "reverse of the amount of noise added" and call it distortion, ok?

I am thinking that i may be using same parameters, but defaults have changed somehow. I understand that preserving signal is not the goal, i just want to make sure that i'm not doing something wrong with the codec. That the data i gathered (and not only me, see link in prev. post) does not in any way mean the files i get with 3.98.3 are lower in quality (hearing quality) than those i get with 3.93.1
Go to the top of the page
+Quote Post
Kohlrabi
post Sep 30 2011, 10:59
Post #21





Group: Super Moderator
Posts: 1046
Joined: 12-March 05
From: Kiel, Germany
Member No.: 20561



The site you linked doesn't contain any data, just an image of two waveforms put onto each other. Please explain what these images are supposed to convey, it's not self-evident to me. Also:

QUOTE
But the visual impression of the waveform comparisons matches actually the results of my (and Jesse's) blind ABX (listening) testings.
What does that even mean?

To say it again, LAME is most certainly not tuned to produce high SNR, but rather to achieve transparency. So, yeah, in a way you can say the drop in SNR going from one version to the other is "intended" because the SNR metric is not important for lossy codec development.

If you really care so much about high SNR, use the old codec.
If you really want to understand the reasons why the SNR dropped in the newer version, study lossy codec development and LAME specifically.

Otherwise this discussion will just go in circles like it does now.

This post has been edited by Kohlrabi: Sep 30 2011, 11:06


--------------------
Ceterum censeo Masterdiskem esse delendam.
Go to the top of the page
+Quote Post
Istrebitel
post Sep 30 2011, 11:04
Post #22





Group: Members
Posts: 54
Joined: 20-October 10
Member No.: 84756



Those images are supposed to convey the same thing that I try to state: SNR dropped in 3.98 compared to 3.93. What you quoted means that blind listening tests confirmed that files made with 3.93 sound better than those with 3.98, i presume.

The discussion goes circles first of all because people dont understand that if something is useless to them, it isnt imperative to be useless to everybody, and try to impose "If i see no point in this, he has no point too, so he must see that i am right and he also has no point in this".

Go to the top of the page
+Quote Post
robert
post Sep 30 2011, 11:06
Post #23


LAME developer


Group: Developer
Posts: 788
Joined: 22-September 01
Member No.: 5



If you call LAME with --verbose command line parameter, you will notice, 3.98 does some scaling of the input data. FWIW, add --scale 1.
Go to the top of the page
+Quote Post
db1989
post Sep 30 2011, 11:23
Post #24





Group: Super Moderator
Posts: 5275
Joined: 23-June 06
Member No.: 32180



QUOTE (Istrebitel @ Sep 30 2011, 11:04) *
Those images are supposed to convey the same thing that I try to state: SNR dropped in 3.98 compared to 3.93.
No. The mean amplitude of the signal generated by mixing the uncompressed and encoded signals, one inverted, increased. pdq pointed out how useless this is as an indication of quality a good number of posts back. Surely you understand the point of psychoacoustics? If the listener can’t hear it and it reduces bitrate, the codec can do whatever it wants to the input audio signal.

QUOTE
What you quoted means that blind listening tests confirmed that files made with 3.93 sound better than those with 3.98, i presume.
I see no ABX logs in the linked thread, or any other information that might enable other listeners to evaluate those posters’ claims. Boring.

QUOTE
The discussion goes circles first of all because people dont understand that if something is useless to them, it isnt imperative to be useless to everybody, and try to impose "If i see no point in this, he has no point too, so he must see that i am right and he also has no point in this".
Again, you are misunderstanding the purpose and mechanics of psychoacoustic modelling, despite ample opportunity to learn how it actually works, despite numerous past threads on the folly of using visual or mathematic indices as any indication of perceived quality, despite ToS8, etc.… Who’s going in circles here, again?
Go to the top of the page
+Quote Post
Kohlrabi
post Sep 30 2011, 11:25
Post #25





Group: Super Moderator
Posts: 1046
Joined: 12-March 05
From: Kiel, Germany
Member No.: 20561



QUOTE (Istrebitel @ Sep 30 2011, 12:04) *
The discussion goes circles first of all because people dont understand that if something is useless to them, it isnt imperative to be useless to everybody, and try to impose "If i see no point in this, he has no point too, so he must see that i am right and he also has no point in this".

You can only expect a meaningful discussion if the participants understand the topic and reasoning. You always state that perceived quality is no concern to you, yet you link to a page where people draw haphazard conclusions from comparing green dots with ABX tests.



--------------------
Ceterum censeo Masterdiskem esse delendam.
Go to the top of the page
+Quote Post

2 Pages V   1 2 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 16th September 2014 - 19:24