IPB

Welcome Guest ( Log In | Register )

2 Pages V   1 2 >  
Reply to this topicStart new topic
RMAA Test on different Codec
master
post Aug 1 2004, 00:17
Post #1





Group: Members
Posts: 180
Joined: 14-January 03
From: Singapore
Member No.: 4566



A user test different codec using RMAA, first saw it HERE.

Here is the list of codecs tested and their parameter.
  • WAV ( original 16/44.1 WAV test signal )
  • MP3 128k ( Cool Edit 2000 1.1 build 2418 )
  • Ogg Vorbis Q6 ( OggEnc v1.0.1, libvorbis 1.0.1, encoding -q6 )
  • AC3 192k ( Vegas 4.0 192 kbps stereo, AC3ACM v0.7 decode )
  • AAC 128k ( iTunes v4.5.0.31 AAC 128 kbps )
  • WMA 128k (Vegas 4.0 128 kbps stereo, Window Media Audio ver.9 )
And HERE is the direct link to his page.

I wonder how Musepack will perform, too bad he didn't include in.


--------------------
Break The Rules!!!
Go to the top of the page
+Quote Post
guruboolez
post Aug 1 2004, 00:22
Post #2





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



Without talking about the complete nonsense of this test, just one comment:

QUOTE
Software Codecs

WAV  ( original 16/44.1 WAV test signal )
MP3 128k ( Cool Edit 2000 1.1 build 2418 )
Ogg Vorbis Q6  ( OggEnc v1.0.1, libvorbis 1.0.1, encoding -q6 )
AC3 192k  ( Vegas 4.0 192 kbps stereo, AC3ACM v0.7 decode )
AAC 128k  ( iTunes v4.5.0.31 AAC 128 kbps )
WMA 128k  (Vegas 4.0 128 kbps stereo, Window Media Audio ver.9 )


I wonder how, or maybe why, this guy is comparing MP3, AAC and WMA at 128 kbps with vorbis at -q6 (190 kbps) rolleyes.gif
Go to the top of the page
+Quote Post
Cygnus X1
post Aug 1 2004, 01:52
Post #3





Group: Members (Donating)
Posts: 676
Joined: 5-June 02
From: New York
Member No.: 2224



WHY don't people understand that perceptual coders can't be evaluated for quality using graphs and numbers? We're not talking about cassette tapes here, people. I just don't get it!! laugh.gif
Go to the top of the page
+Quote Post
Peter
post Aug 1 2004, 02:04
Post #4


foobar2000 developer


Group: Admin
Posts: 3292
Joined: 30-September 01
Member No.: 84



This has been discussed before: http://www.hydrogenaudio.org/forums/index....showtopic=4516&
Go to the top of the page
+Quote Post
master
post Aug 1 2004, 08:06
Post #5





Group: Members
Posts: 180
Joined: 14-January 03
From: Singapore
Member No.: 4566



QUOTE (Cygnus X1 @ Jul 31 2004, 04:52 PM)
WHY don't people understand that perceptual coders can't be evaluated for quality using graphs and numbers? We're not talking about cassette tapes here, people. I just don't get it!!  laugh.gif
*

Why not, mind to explain?

My thinking is straight forward. What is the objective of the codec? Compress and decompress and ensure the signal reproduce close to original, isn't it?


--------------------
Break The Rules!!!
Go to the top of the page
+Quote Post
kjoonlee
post Aug 1 2004, 08:16
Post #6





Group: Members
Posts: 2526
Joined: 25-July 02
From: South Korea
Member No.: 2782



Not true. It's to compress and decompress and ensure that you can't tell the original and the compressed sample apart with your ears, not your eyes.

This post has been edited by kjoonlee: Aug 1 2004, 09:09


--------------------
http://blacksun.ivyro.net/vorbis/vorbisfaq.htm
Go to the top of the page
+Quote Post
master
post Aug 1 2004, 09:36
Post #7





Group: Members
Posts: 180
Joined: 14-January 03
From: Singapore
Member No.: 4566



QUOTE (kjoonlee @ Jul 31 2004, 11:16 PM)
Not true. It's to compress and decompress and ensure that you can't tell the original and the compressed sample apart with your ears, not your eyes.
*

But hearing is subjective, measurement is objective. Thus the measurement comes in.

And what do you mean by can't tell with original and the compressed sample apart? Isn't that means to reproduce a signal close to original signal?


--------------------
Break The Rules!!!
Go to the top of the page
+Quote Post
kjoonlee
post Aug 1 2004, 09:41
Post #8





Group: Members
Posts: 2526
Joined: 25-July 02
From: South Korea
Member No.: 2782



Sound quality is subjective, so you can't rely on somebody else's measurements.

No, you can distort the signal on purpose to get smaller filesizes, and it can still sound the same to your ears. That's lossy compression for you.

This post has been edited by kjoonlee: Aug 1 2004, 09:45


--------------------
http://blacksun.ivyro.net/vorbis/vorbisfaq.htm
Go to the top of the page
+Quote Post
Cygnus X1
post Aug 1 2004, 15:43
Post #9





Group: Members (Donating)
Posts: 676
Joined: 5-June 02
From: New York
Member No.: 2224



QUOTE (master @ Aug 1 2004, 02:06 AM)
QUOTE (Cygnus X1 @ Jul 31 2004, 04:52 PM)
WHY don't people understand that perceptual coders can't be evaluated for quality using graphs and numbers? We're not talking about cassette tapes here, people. I just don't get it!!  laugh.gif
*

Why not, mind to explain?

My thinking is straight forward. What is the objective of the codec? Compress and decompress and ensure the signal reproduce close to original, isn't it?
*



Sure. It's been demonstrated time and time again on this and other boards that the correlation between the way a signal looks on a graph and how it actually sounds is not as high as you might expect. For example, Microsoft did a campaign a few years back that compared it's Window media format to other formats by asserting that WMA "throws away" less of the .wav file, and even had difference files to show what the encoder had chucked. Well, guess what? WMA at 96kbps happened to sound the worst of all the formats tested once you listened to it, pretty graphs and difference files aside.

Another example is the infamous BLADE codec, which looks nice on a frequency graph because it lacked a lowpass, but sounded like a lawnmower running over a chipmunk once you listened to it. Things like SNR, crosstalk, and frequency response curves are NOT good indicators of a lossy codec's quality. Even MP3 at 96kbps probably has a high SNR, low crosstalk and THD, and (in the case of a bad codec like BLADE) a nice frequency response curve. Trouble is, since when has 96kbps MP3 sounded good? Trust you ears, not your eyes. If you want to be objective about sound quality, a double-blind ABX test is the way to go.

The point of lossy compression is to obtain an accurate-sounding representation of the original signal at a fraction of the size. If you are looking for something that is absolutely true to the source, lossless is a better choice for you.

This post has been edited by Cygnus X1: Aug 1 2004, 15:44
Go to the top of the page
+Quote Post
Pio2001
post Aug 1 2004, 15:56
Post #10


Moderator


Group: Super Moderator
Posts: 3936
Joined: 29-September 01
Member No.: 73



See for example JohnV's explanation in the 5th message of this page :
http://de.europe.creative.com/support/foru...&foru=12&page=2
Go to the top of the page
+Quote Post
master
post Aug 1 2004, 15:57
Post #11





Group: Members
Posts: 180
Joined: 14-January 03
From: Singapore
Member No.: 4566



QUOTE (kjoonlee @ Aug 1 2004, 12:41 AM)
Sound quality is subjective, so you can't rely on somebody else's measurements.

No, you can distort the signal on purpose to get smaller filesizes, and it can still sound the same to your ears. That's lossy compression for you.
*

Sound quality ain't subjective, personal preference is subjective. Please....

And this is not somebody else measurement, this is scientific measurement.

And, no, you don't distort the signal on purpose to get smaller filesizes, but you drop some bit and squeeze them to get smaller size, and during decoding you get the bit back by mathematic calculation. When there is a miss bit or bit error, that is when distortion and other symptoms occured. Or am I misunderstand how the codec works?


[Edited] Add last sentance.

This post has been edited by master: Aug 1 2004, 16:01


--------------------
Break The Rules!!!
Go to the top of the page
+Quote Post
Garf
post Aug 1 2004, 16:06
Post #12


Server Admin


Group: Admin
Posts: 4886
Joined: 24-September 01
Member No.: 13



He meant the same thing as you are saying.

A codec uses a psychoacoustic model to determine how "inaccurate" something can be stored (less accurate->less bits) and still sound the same. You can't judge how well this is done without having a perfect psychoacoustic model yourself (judging one model with another is flawed, too). The best thing you do have is your ears.

There's some great explanations posted why this kind of "scientific" analysis you're doing is useless and flawed, please read them.

This post has been edited by Garf: Aug 1 2004, 16:08
Go to the top of the page
+Quote Post
master
post Aug 1 2004, 16:15
Post #13





Group: Members
Posts: 180
Joined: 14-January 03
From: Singapore
Member No.: 4566



QUOTE (Cygnus X1 @ Aug 1 2004, 06:43 AM)
Sure. It's been demonstrated time and time again on this and other boards that the correlation between the way a signal looks on a graph and how it actually sounds is not as high as you might expect. For example, Microsoft did a campaign a few years back that compared it's Window media format to other formats by asserting that WMA "throws away" less of the .wav file, and even had difference files to show what the encoder had chucked. Well, guess what? WMA at 96kbps happened to sound the worst of all the formats tested once you listened to it, pretty graphs and difference files aside.

Another example is the infamous BLADE codec, which looks nice on a frequency graph because it lacked a lowpass, but sounded like a lawnmower running over a chipmunk once you listened to it.  Things like SNR, crosstalk, and frequency response curves are NOT good indicators of a lossy codec's quality. Even MP3 at 96kbps probably has a high SNR, low crosstalk and THD, and (in the case of a bad codec like BLADE) a nice frequency response curve. Trouble is, since when has 96kbps MP3 sounded good? Trust you ears, not your eyes. If you want to be objective about sound quality, a double-blind ABX test is the way to go.

The point of lossy compression is to obtain an accurate-sounding representation of the original signal at a fraction of the size. If you are looking for something that is absolutely true to the source, lossless is a better choice for you.
*

1. Do you have the graph for the BLADE and WMA @ 96kbps? Appreciate if you can show them to me. May be a frequency sweep?

2. I trust my ears, and I trust the measurement too. Some people tend to prefer some harmonic coz it is easy on their ears, some are not. That is subjective. I may prefer something with higher harmonic, but that does not means the sound quality is the best. How we define sound quality here? Fidelity, right?

3. I agree with your statement:"The point of lossy compression is to obtain an accurate-sounding representation of the original signal at a fraction of the size." That said, a codec job is still doing sound reproduction job which close to original as much as possible in a decent file size.

4. No no, this is nothing to do with lossless. Just a discussion on lossy format.


[Edited] Remove unnecessary quoting.

This post has been edited by master: Aug 1 2004, 16:15


--------------------
Break The Rules!!!
Go to the top of the page
+Quote Post
markanini
post Aug 1 2004, 16:18
Post #14





Group: Members
Posts: 551
Joined: 22-December 03
From: Malmö, Sweden
Member No.: 10615



QUOTE
...and squeeze them to get smaller size

That didnt sound very scientific.

"Dropping some bits" does mean adding distorsion.
Go to the top of the page
+Quote Post
master
post Aug 1 2004, 16:20
Post #15





Group: Members
Posts: 180
Joined: 14-January 03
From: Singapore
Member No.: 4566



QUOTE (Pio2001 @ Aug 1 2004, 06:56 AM)
See for example JohnV's explanation in the 5th message of this page :
http://de.europe.creative.com/support/foru...&foru=12&page=2
*

Thanks for the link!!!

I promised you I will read it (I already bookmark it). wink.gif


--------------------
Break The Rules!!!
Go to the top of the page
+Quote Post
master
post Aug 1 2004, 16:29
Post #16





Group: Members
Posts: 180
Joined: 14-January 03
From: Singapore
Member No.: 4566



QUOTE (markanini @ Aug 1 2004, 07:18 AM)
That didnt sound very scientific.

"Dropping some bits" does mean adding distorsion.
*

Why not scientific??? How you drop the bit? How you calculate back the bit? This is mathematic calculation and it is scientific. Am I wrong again?

Only during decoding when the calculation has a miss bit or bit error, then distortion occur. If you get the bit correct, no distortion should be happened.


[Edited] Remove unnecessary quoting.... again.

This post has been edited by master: Aug 1 2004, 16:31


--------------------
Break The Rules!!!
Go to the top of the page
+Quote Post
Pio2001
post Aug 1 2004, 16:31
Post #17


Moderator


Group: Super Moderator
Posts: 3936
Joined: 29-September 01
Member No.: 73



QUOTE (master @ Aug 1 2004, 04:20 PM)
Thanks for the link!!!


Just a remark after having read the link again : I think that JohnV uses improperly the term "quantization noise". "Encoding noise" or just "noise" should fit better. Besides this wording issue, the rest of the message is correct.
Go to the top of the page
+Quote Post
master
post Aug 1 2004, 16:36
Post #18





Group: Members
Posts: 180
Joined: 14-January 03
From: Singapore
Member No.: 4566



QUOTE (Garf @ Aug 1 2004, 07:06 AM)
He meant the same thing as you are saying.

A codec uses a psychoacoustic model to determine how "inaccurate" something can be stored (less accurate->less bits) and still sound the same. You can't judge how well this is done without having a perfect psychoacoustic model yourself (judging one model with another is flawed, too). The best thing you do have is your ears.

There's some great explanations posted why this kind of "scientific" analysis you're doing is useless and flawed, please read them.
*

Mind to explain further on psychoacoustic model?

Personally, I think the best judgement is by Audio Precesion measurement equipment. What do you think?

And no, I am not the one who do the testing. I just happened to saw the link at RMAA Forum.


[Edited] Add last paragraph.

This post has been edited by master: Aug 1 2004, 16:39


--------------------
Break The Rules!!!
Go to the top of the page
+Quote Post
Garf
post Aug 1 2004, 16:42
Post #19


Server Admin


Group: Admin
Posts: 4886
Joined: 24-September 01
Member No.: 13



QUOTE (master @ Aug 1 2004, 05:29 PM)
Why not scientific??? How you drop the bit? How you calculate back the bit? This is mathematic calculation and it is scientific. Am I wrong again?

Only during decoding when the calculation has a miss bit or bit error, then distortion occur. If you get the bit correct, no distortion should be happened.


Example (that's not that far from what really happens):

I need to store 11, 52, 3, 93, 39, 87

and I store 1, 5, 0, 9, 4, 9 instead. (divide by 10 and only store the integer part)

Upon decoding I get 10, 50, 0, 90, 40, 90.

The difference between what I have and what I started with is the quantization noise (which is distortion).

This post has been edited by Garf: Aug 1 2004, 16:42
Go to the top of the page
+Quote Post
Garf
post Aug 1 2004, 16:44
Post #20


Server Admin


Group: Admin
Posts: 4886
Joined: 24-September 01
Member No.: 13



QUOTE
Mind to explain further on psychoacoustic model?

Personally, I think the best judgement is by Audio Precesion measurement equipment. What do you think?


The links above contain information on psychoacoustic models.

"Audio Precesion measurement equipment" cannot determine what is *audible* distortion as is hence completely useless.

This post has been edited by Garf: Aug 1 2004, 16:44
Go to the top of the page
+Quote Post
master
post Aug 1 2004, 18:16
Post #21





Group: Members
Posts: 180
Joined: 14-January 03
From: Singapore
Member No.: 4566



QUOTE (Garf @ Aug 1 2004, 07:44 AM)
The links above contain information on psychoacoustic models.

"Audio Precesion measurement equipment" cannot determine what is *audible* distortion as is hence completely useless.
*

Hmm.... I am afraid I cannot agree with you on this.

AP did measure distortion, it even measured THD+N and it measured those "audible" and "unaudible" (to human ears) distortion.


[Edited] Missing word.

This post has been edited by master: Aug 1 2004, 18:17


--------------------
Break The Rules!!!
Go to the top of the page
+Quote Post
Garf
post Aug 1 2004, 18:27
Post #22


Server Admin


Group: Admin
Posts: 4886
Joined: 24-September 01
Member No.: 13



Things like RMAA measure mathematical distortions of a sine wave.

Now:

1) Distortion in the audible range is *not* the same as (the amount of) audible distortion.
2) The numbers don't tell you how audible something is (want an IMD of 0.01 or 0.001? How bad is that?)
3) It could only mean something when you listen to pure sine signals only. Which no sane person does.

Again: completely useless.

This post has been edited by Garf: Aug 1 2004, 18:32
Go to the top of the page
+Quote Post
Gabriel
post Aug 1 2004, 18:50
Post #23


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



The purpose of a psychoacoustic codec is to introduce as much (re)quantization noise as possible, while maintaining a good sound quality to your ears.

This is because reducing the number of bits to store numbers is equivalent to introducing noise.
Go to the top of the page
+Quote Post
Pio2001
post Aug 1 2004, 21:41
Post #24


Moderator


Group: Super Moderator
Posts: 3936
Joined: 29-September 01
Member No.: 73



Master's arguments are not illogical. Studies have been made that show how much THD people can hear (it must certainly be somewhere in the AES journal).
The point is elsewhere : a lossy codec removes the parts of the music that the psychoacoustic model finds the less audible.
Go to http://www.mp3-tech.org/ , in the menu, click "MP3", then "ovberview of the MP3 techniques".
This can be harmonics of an instruments, parts of sounds of other instruments after the attack of a percussion, some instruments that can't be heard during percussive sounds... The way lossy codec change the sound can appear completely chaotic...
More important, a codec begins its work of removing infos with its psychoacoustic model when very complex sound begin to appear. With RMAA test tones, only the roughest part of the model is at work (absolute threshold of hearing + frequency masking). Temporal masking, joint stereo tuning, choice between long and short blocks, and all the deep stuff, whose quality is capital for the result, is not at work on RMAA test tones. Thus the results can't give an idea of the real performances of the codec.
Go to the top of the page
+Quote Post
master
post Aug 2 2004, 12:05
Post #25





Group: Members
Posts: 180
Joined: 14-January 03
From: Singapore
Member No.: 4566



Ok, after reading the link, and some discussion with my friend, more or less I get the picture of it. Thanks again. wink.gif

Now, the "inaudible" noise you mentioned here, from what I think, is truly subjective. Reason? Everyone has different ears.

When our ears are different from one to another, the ability to capture the quantization noise from the product of psychoacoustic model is also different.

Hmm.... so what is the best measurement tool here? While AP cannot measure how well the quantization noise handling by the codec, can it paint a overall behaviour picture by doing a frequency sweep?

Open for discussion, truly appreciate to those reply to me and clear my doubts. smile.gif


[Edited] Typo in bold.

This post has been edited by master: Aug 2 2004, 12:07


--------------------
Break The Rules!!!
Go to the top of the page
+Quote Post

2 Pages V   1 2 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 20th October 2014 - 13:26