Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: RMAA Test on different Codec (Read 11837 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

RMAA Test on different Codec

A user test different codec using RMAA, first saw it HERE.

Here is the list of codecs tested and their parameter.
  • WAV  ( original 16/44.1 WAV test signal )
  • MP3 128k ( Cool Edit 2000 1.1 build 2418 )
  • Ogg Vorbis Q6  ( OggEnc v1.0.1, libvorbis 1.0.1, encoding -q6 )
  • AC3 192k  ( Vegas 4.0 192 kbps stereo, AC3ACM v0.7 decode )
  • AAC 128k  ( iTunes v4.5.0.31 AAC 128 kbps )
  • WMA 128k  (Vegas 4.0 128 kbps stereo, Window Media Audio ver.9 )
And HERE is the direct link to his page.

I wonder how Musepack will perform, too bad he didn't include in.
Break The Rules!!!

RMAA Test on different Codec

Reply #1
Without talking about the complete nonsense of this test, just one comment:

Quote
Software Codecs

WAV  ( original 16/44.1 WAV test signal )
MP3 128k ( Cool Edit 2000 1.1 build 2418 )
Ogg Vorbis Q6  ( OggEnc v1.0.1, libvorbis 1.0.1, encoding -q6 )
AC3 192k  ( Vegas 4.0 192 kbps stereo, AC3ACM v0.7 decode )
AAC 128k  ( iTunes v4.5.0.31 AAC 128 kbps )
WMA 128k  (Vegas 4.0 128 kbps stereo, Window Media Audio ver.9 )


I wonder how, or maybe why, this guy is comparing MP3, AAC and WMA at 128 kbps with vorbis at -q6 (190 kbps)

RMAA Test on different Codec

Reply #2
WHY don't people understand that perceptual coders can't be evaluated for quality using graphs and numbers? We're not talking about cassette tapes here, people. I just don't get it!! 


RMAA Test on different Codec

Reply #4
Quote
WHY don't people understand that perceptual coders can't be evaluated for quality using graphs and numbers? We're not talking about cassette tapes here, people. I just don't get it!! 
[a href="index.php?act=findpost&pid=230705"][{POST_SNAPBACK}][/a]

Why not, mind to explain?

My thinking is straight forward. What is the objective of the codec? Compress and decompress and ensure the signal reproduce close to original, isn't it?
Break The Rules!!!

RMAA Test on different Codec

Reply #5
Not true. It's to compress and decompress and ensure that you can't tell the original and the compressed sample apart with your ears, not your eyes.

RMAA Test on different Codec

Reply #6
Quote
Not true. It's to compress and decompress and ensure that you can't tell the original and the compressed sample apart with your ears, not your eyes.
[a href="index.php?act=findpost&pid=230760"][{POST_SNAPBACK}][/a]

But hearing is subjective, measurement is objective. Thus the measurement comes in.

And what do you mean by can't tell with original and the compressed sample apart? Isn't that means to reproduce a signal close to original signal?
Break The Rules!!!

RMAA Test on different Codec

Reply #7
Sound quality is subjective, so you can't rely on somebody else's measurements.

No, you can distort the signal on purpose to get smaller filesizes, and it can still sound the same to your ears. That's lossy compression for you.

RMAA Test on different Codec

Reply #8
Quote
Quote
WHY don't people understand that perceptual coders can't be evaluated for quality using graphs and numbers? We're not talking about cassette tapes here, people. I just don't get it!! 
[a href="index.php?act=findpost&pid=230705"][{POST_SNAPBACK}][/a]

Why not, mind to explain?

My thinking is straight forward. What is the objective of the codec? Compress and decompress and ensure the signal reproduce close to original, isn't it?
[a href="index.php?act=findpost&pid=230758"][{POST_SNAPBACK}][/a]


Sure. It's been demonstrated time and time again on this and other boards that the correlation between the way a signal looks on a graph and how it actually sounds is not as high as you might expect. For example, Microsoft did a campaign a few years back that compared it's Window media format to other formats by asserting that WMA "throws away" less of the .wav file, and even had difference files to show what the encoder had chucked. Well, guess what? WMA at 96kbps happened to sound the worst of all the formats tested once you listened to it, pretty graphs and difference files aside.

Another example is the infamous BLADE codec, which looks nice on a frequency graph because it lacked a lowpass, but sounded like a lawnmower running over a chipmunk once you listened to it.  Things like SNR, crosstalk, and frequency response curves are NOT good indicators of a lossy codec's quality. Even MP3 at 96kbps probably has a high SNR, low crosstalk and THD, and (in the case of a bad codec like BLADE) a nice frequency response curve. Trouble is, since when has 96kbps MP3 sounded good? Trust you ears, not your eyes. If you want to be objective about sound quality, a double-blind ABX test is the way to go.

The point of lossy compression is to obtain an accurate-sounding representation of the original signal at a fraction of the size. If you are looking for something that is absolutely true to the source, lossless is a better choice for you.


RMAA Test on different Codec

Reply #10
Quote
Sound quality is subjective, so you can't rely on somebody else's measurements.

No, you can distort the signal on purpose to get smaller filesizes, and it can still sound the same to your ears. That's lossy compression for you.
[a href="index.php?act=findpost&pid=230771"][{POST_SNAPBACK}][/a]

Sound quality ain't subjective, personal preference is subjective. Please....

And this is not somebody else measurement, this is scientific measurement.

And, no, you don't distort the signal on purpose to get smaller filesizes, but you drop some bit and squeeze them to get smaller size, and during decoding you get the bit back by mathematic calculation. When there is a miss bit or bit error, that is when distortion and other symptoms occured. Or am I misunderstand how the codec works?


[Edited] Add last sentance.
Break The Rules!!!

RMAA Test on different Codec

Reply #11
He meant the same thing as you are saying.

A codec uses a psychoacoustic model to determine how "inaccurate" something can be stored (less accurate->less bits) and still sound the same. You can't judge how well this is done without having a perfect psychoacoustic model yourself (judging one model with another is flawed, too). The best thing you do have is your ears.

There's some great explanations posted why this kind of "scientific" analysis you're doing is useless and flawed, please read them.

RMAA Test on different Codec

Reply #12
Quote
Sure. It's been demonstrated time and time again on this and other boards that the correlation between the way a signal looks on a graph and how it actually sounds is not as high as you might expect. For example, Microsoft did a campaign a few years back that compared it's Window media format to other formats by asserting that WMA "throws away" less of the .wav file, and even had difference files to show what the encoder had chucked. Well, guess what? WMA at 96kbps happened to sound the worst of all the formats tested once you listened to it, pretty graphs and difference files aside.

Another example is the infamous BLADE codec, which looks nice on a frequency graph because it lacked a lowpass, but sounded like a lawnmower running over a chipmunk once you listened to it.  Things like SNR, crosstalk, and frequency response curves are NOT good indicators of a lossy codec's quality. Even MP3 at 96kbps probably has a high SNR, low crosstalk and THD, and (in the case of a bad codec like BLADE) a nice frequency response curve. Trouble is, since when has 96kbps MP3 sounded good? Trust you ears, not your eyes. If you want to be objective about sound quality, a double-blind ABX test is the way to go.

The point of lossy compression is to obtain an accurate-sounding representation of the original signal at a fraction of the size. If you are looking for something that is absolutely true to the source, lossless is a better choice for you.
[a href="index.php?act=findpost&pid=230841"][{POST_SNAPBACK}][/a]

1. Do you have the graph for the BLADE and WMA @ 96kbps? Appreciate if you can show them to me. May be a frequency sweep?

2. I trust my ears, and I trust the measurement too. Some people tend to prefer some harmonic coz it is easy on their ears, some are not. That is subjective. I may prefer something with higher harmonic, but that does not means the sound quality is the best. How we define sound quality here? Fidelity, right?

3. I agree with your statement:"The point of lossy compression is to obtain an accurate-sounding representation of the original signal at a fraction of the size." That said, a codec job is still doing sound reproduction job which close to original as much as possible in a decent file size.

4. No no, this is nothing to do with lossless. Just a discussion on lossy format.


[Edited] Remove unnecessary quoting.
Break The Rules!!!

RMAA Test on different Codec

Reply #13
Quote
...and squeeze them to get smaller size

That didnt sound very scientific.

"Dropping some bits" does mean adding distorsion.

RMAA Test on different Codec

Reply #14
Quote
See for example JohnV's explanation in the 5th message of this page :
http://de.europe.creative.com/support/foru...&foru=12&page=2
[a href="index.php?act=findpost&pid=230847"][{POST_SNAPBACK}][/a]

Thanks for the link!!!

I promised you I will read it (I already bookmark it). 
Break The Rules!!!

RMAA Test on different Codec

Reply #15
Quote
That didnt sound very scientific.

"Dropping some bits" does mean adding distorsion.
[a href="index.php?act=findpost&pid=230856"][{POST_SNAPBACK}][/a]

Why not scientific??? How you drop the bit? How you calculate back the bit? This is mathematic calculation and it is scientific. Am I wrong again?

Only during decoding when the calculation has a miss bit or bit error, then distortion occur. If you get the bit correct, no distortion should be happened.


[Edited] Remove unnecessary quoting.... again.
Break The Rules!!!

RMAA Test on different Codec

Reply #16
Quote
Thanks for the link!!!


Just a remark after having read the link again : I think that JohnV uses improperly the term "quantization noise". "Encoding noise" or just "noise" should fit better. Besides this wording issue, the rest of the message is correct.

RMAA Test on different Codec

Reply #17
Quote
He meant the same thing as you are saying.

A codec uses a psychoacoustic model to determine how "inaccurate" something can be stored (less accurate->less bits) and still sound the same. You can't judge how well this is done without having a perfect psychoacoustic model yourself (judging one model with another is flawed, too). The best thing you do have is your ears.

There's some great explanations posted why this kind of "scientific" analysis you're doing is useless and flawed, please read them.
[a href="index.php?act=findpost&pid=230851"][{POST_SNAPBACK}][/a]

Mind to explain further on psychoacoustic model?

Personally, I think the best judgement is by Audio Precesion measurement equipment. What do you think?

And no, I am not the one who do the testing. I just happened to saw the link at RMAA Forum.


[Edited] Add last paragraph.
Break The Rules!!!

RMAA Test on different Codec

Reply #18
Quote
Why not scientific??? How you drop the bit? How you calculate back the bit? This is mathematic calculation and it is scientific. Am I wrong again?

Only during decoding when the calculation has a miss bit or bit error, then distortion occur. If you get the bit correct, no distortion should be happened.


Example (that's not that far from what really happens):

I need to store 11, 52, 3, 93, 39, 87

and I store 1, 5, 0, 9, 4, 9 instead. (divide by 10 and only store the integer part)

Upon decoding I get 10, 50, 0, 90, 40, 90.

The difference between what I have and what I started with is the quantization noise (which is distortion).

 

RMAA Test on different Codec

Reply #19
Quote
Mind to explain further on psychoacoustic model?

Personally, I think the best judgement is by Audio Precesion measurement equipment. What do you think?


The links above contain information on psychoacoustic models.

"Audio Precesion measurement equipment" cannot determine what is *audible* distortion as is hence completely useless.

RMAA Test on different Codec

Reply #20
Quote
The links above contain information on psychoacoustic models.

"Audio Precesion measurement equipment" cannot determine what is *audible* distortion as is hence completely useless.
[a href="index.php?act=findpost&pid=230870"][{POST_SNAPBACK}][/a]

Hmm.... I am afraid I cannot agree with you on this.

AP did measure distortion, it even measured THD+N and it measured those "audible" and "unaudible" (to human ears) distortion.


[Edited] Missing word.
Break The Rules!!!

RMAA Test on different Codec

Reply #21
Things like RMAA measure mathematical distortions of a sine wave.

Now:

1) Distortion in the audible range is *not* the same as (the amount of) audible distortion.
2) The numbers don't tell you how audible something is (want an IMD of 0.01 or 0.001? How bad is that?)
3) It could only mean something when you listen to pure sine signals only. Which no sane person does.

Again: completely useless.

RMAA Test on different Codec

Reply #22
The purpose of a psychoacoustic codec is to introduce as much (re)quantization noise as possible, while maintaining a good sound quality to your ears.

This is because reducing the number of bits to store numbers is equivalent to introducing noise.

RMAA Test on different Codec

Reply #23
Master's arguments are not illogical. Studies have been made that show how much THD people can hear (it must certainly be somewhere in the AES journal).
The point is elsewhere : a lossy codec removes the parts of the music that the psychoacoustic model finds the less audible.
Go to http://www.mp3-tech.org/ , in the menu, click "MP3", then "ovberview of the MP3 techniques".
This can be harmonics of an instruments, parts of sounds of other instruments after the attack of a percussion, some instruments that can't be heard during percussive sounds... The way lossy codec change the sound can appear completely chaotic...
More important, a codec begins its work of removing infos with its psychoacoustic model when very complex sound begin to appear. With RMAA test tones, only the roughest part of the model is at work (absolute threshold of hearing + frequency masking). Temporal masking, joint stereo tuning, choice between long and short blocks, and all the deep stuff, whose quality is capital for the result, is not at work on RMAA test tones. Thus the results can't give an idea of the real performances of the codec.

RMAA Test on different Codec

Reply #24
Ok, after reading the link, and some discussion with my friend, more or less I get the picture of it. Thanks again. 

Now, the "inaudible" noise you mentioned here, from what I think, is truly subjective. Reason? Everyone has different ears.

When our ears are different from one to another, the ability to capture the quantization noise from the product of psychoacoustic model is also different.

Hmm.... so what is the best measurement tool here? While AP cannot measure how well the quantization noise handling by the codec, can it paint a overall behaviour picture by doing a frequency sweep?

Open for discussion, truly appreciate to those reply to me and clear my doubts.


[Edited] Typo in bold.
Break The Rules!!!