IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
Does it sense to listening test of lossy codecs at higher bitrates?
Anakunda
post Aug 16 2012, 08:54
Post #1





Group: Members
Posts: 484
Joined: 24-November 08
Member No.: 63072



HI!
Does it sense perform a listening tests of lossy codecs at higher bitrates (192k and higher) as many of them are already considered as "transparent"?
If the listening test won't show the difference between various codecs and encoder profiles, are the spectral graphs the only measure of the audio quality (instead doing a listening test)? And what should be important by judging the audio data quality from spectral view? Thanks rolleyes.gif
Go to the top of the page
+Quote Post
knutinh
post Aug 16 2012, 09:38
Post #2





Group: Members
Posts: 570
Joined: 1-November 06
Member No.: 37047



QUOTE (Anakunda @ Aug 16 2012, 09:54) *
HI!
Does it sense perform a listening tests of lossy codecs at higher bitrates (192k and higher) as many of them are already considered as "transparent"?

Perhaps, but as the audible difference approach zero, the effort needed to separate good ones from lesser ones approach infinite.
QUOTE
If the listening test won't show the difference between various codecs and encoder profiles, are the spectral graphs the only measure of the audio quality (instead doing a listening test)? And what should be important by judging the audio data quality from spectral view? Thanks rolleyes.gif

You should not judge the quality of a lossy audio codec from some spectral view. Lossy audio codecs are built to satisfy human listening, and that alone. If human listening can detect no flaws, then all is good.

-k
Go to the top of the page
+Quote Post
Garf
post Aug 16 2012, 10:26
Post #3


Server Admin


Group: Admin
Posts: 4886
Joined: 24-September 01
Member No.: 13



QUOTE (Anakunda @ Aug 16 2012, 09:54) *
HI!
Does it sense perform a listening tests of lossy codecs at higher bitrates (192k and higher) as many of them are already considered as "transparent"?


The last public MP3 listening test here had all codecs statistically tied at 128kbps. There's no reason to believe >128kbps and newer codecs could produce a different result, on the contrary.

QUOTE
If the listening test won't show the difference between various codecs and encoder profiles, are the spectral graphs the only measure of the audio quality (instead doing a listening test)? And what should be important by judging the audio data quality from spectral view? Thanks rolleyes.gif


If the listening test tells you that there's no difference, then that is the result. You can't judge the quality of an audio codec by looking at images.

The only other thing that you can do is to try to make your test more sensitive, by for example using trained listeners (but you result won't reflect the general population any more) or having more listeners and samples.

For the tests conducted here, the general observation is that the harder the test, the more difficult it is to get people to send in results, which is why nobody has tried high bitrate tests since.

This post has been edited by Garf: Aug 16 2012, 10:31
Go to the top of the page
+Quote Post
Anakunda
post Aug 16 2012, 10:58
Post #4





Group: Members
Posts: 484
Joined: 24-November 08
Member No.: 63072



Yep I know, the ears are heavily imperfect so ears don't really know what is perfect and what is bilked. So there's no rely on the listening experience and only valuable assessment of resulting audio quality can be analysis, a relevant key to determine the quality from spectral graphs or another numerical based analysis of the audio data, something like that?
Go to the top of the page
+Quote Post
halb27
post Aug 16 2012, 11:13
Post #5





Group: Members
Posts: 2448
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



A usual listening test at high bitrate doesn't make sense for the reasons mentioned.

Getting a feeling however about the amount codecs can fail occasionally even at very high bitrate is worthwhile IMO. This means collecting problem samples for various codecs and finding out how bad various people judge them.



--------------------
lame3100m -V1 --insane-factor 0.75
Go to the top of the page
+Quote Post
greynol
post Aug 16 2012, 12:28
Post #6





Group: Super Moderator
Posts: 10345
Joined: 1-April 04
From: San Francisco
Member No.: 13167



QUOTE (Anakunda @ Aug 16 2012, 02:58) *
Yep I know, the ears are heavily imperfect so ears don't really know what is perfect and what is bilked.

If the ear couldn't be fooled, then there would never have been these codecs in the first place; at any bitrate.

QUOTE
only valuable assessment of resulting audio quality can be analysis

WRONG! There is no valuable assessment other than double-blind listening tests regardless of the bitrate. No matter how many times you present this notion that there is an alternate means to judge perceptual codecs, the answer is the same: the notion is faulty.


--------------------
Breath is found in plots and DR figures.
Go to the top of the page
+Quote Post
Garf
post Aug 16 2012, 12:52
Post #7


Server Admin


Group: Admin
Posts: 4886
Joined: 24-September 01
Member No.: 13



QUOTE (Anakunda @ Aug 16 2012, 11:58) *
Yep I know, the ears are heavily imperfect so ears don't really know what is perfect and what is bilked.


This sentence is 100% meaningless. What could the "perfect" you speak of possibly even mean?
Go to the top of the page
+Quote Post
db1989
post Aug 16 2012, 12:54
Post #8





Group: Super Moderator
Posts: 5275
Joined: 23-June 06
Member No.: 32180



QUOTE (Anakunda @ Aug 16 2012, 10:58) *
So there's no rely on the listening experience and only valuable assessment of resulting audio quality can be analysis, a relevant key to determine the quality from spectral graphs or another numerical based analysis of the audio data, something like that?
what
Go to the top of the page
+Quote Post
chrisz78
post Aug 16 2012, 12:56
Post #9





Group: Validating
Posts: 6
Joined: 23-May 12
Member No.: 100095



QUOTE (greynol @ Aug 16 2012, 13:28) *
WRONG! There is no valuable assessment other than double-blind listening tests regardless of the bitrate. No matter how many times you present this notion that there is an alternate means to judge perceptual codecs, the answer is the same: the notion is faulty.


To me, the most reliable assessment of any signal conversion is still to subtract the output from the input, look at how large the residuals (i.e. the things lost in the processing) are in comparison to the original signal, and whether the residuals sound recognizably like the music encoded (which is bad, as in that case recognizable parts of the original recording are distorted or left out of the encoding). As you raise the bitrate - no matter if dealing with MP3, AAC, or other compression scheme - the residuals get smaller and less "music-like". If you are getting residuals that are far below the signal level at any moment and frequency, and thus always sure to be masked by the signal itself, you can safely say that this or that codec will be transparent for this or that kind of input signal, and you are not just lucky that the combination of your particular playback equipment and your own pair of ears does not notice any audible difference: Some artifacts or losses only become audible with certain kinds of speakers or headphones that accidentally happen to have a peak in their response exactly at the affected frequencies so the smallish actual error of the codec sticks out like the proverbial sore thumb. This can be a really bad surprise if you upgrade your speakers and suddenly find your older encodes unlistenable - so much for listening being the "only reliable way" to judge a lossy encoding!

Of course, results will vary with different input material: Some of the most difficult things to encode are e.g. inherently crackly analogue recordings sourced from vinyl or 78rpm discs that have not been denoised/declicked before encoding, or recordings that contain non-steady broadband noises as part of the desired sound (e.g. "nature sounds" like wind, rain, surf, trains, frying-pans etc. as they all do occur occasionally in movie soundtracks or audio plays / radio drama where you would usually expect that narrow-bandwidth "spoken word" settings might do the job). Settings that provide perfectly good sound with cleanly recorded music of any genre might not be sufficient for these kinds of sound, while OTOH slow-moving music with few transients and percussive sounds might still sound fine at a far smaller bandwidth.
Go to the top of the page
+Quote Post
greynol
post Aug 16 2012, 13:16
Post #10





Group: Super Moderator
Posts: 10345
Joined: 1-April 04
From: San Francisco
Member No.: 13167



Difference signals are just as useless as spectral plots. There is no amount of theorizing that will change this for either.


--------------------
Breath is found in plots and DR figures.
Go to the top of the page
+Quote Post
knutinh
post Aug 16 2012, 13:34
Post #11





Group: Members
Posts: 570
Joined: 1-November 06
Member No.: 37047



QUOTE (Anakunda @ Aug 16 2012, 11:58) *
Yep I know, the ears are heavily imperfect so ears don't really know what is perfect and what is bilked. So there's no rely on the listening experience and only valuable assessment of resulting audio quality can be analysis, a relevant key to determine the quality from spectral graphs or another numerical based analysis of the audio data, something like that?

Think of it like this: the main goal of a toothbrush is to make your teeth clean. Now, if most brands of toothbrushes were able to accomplish perfect clean teeth, would you choose brands based on which toothbrush was best at cleaning other parts of your body? Of course not, a toothbrush is for cleaning teeth, and that is the main performance characteristic one should use to benchmark them (of course price, attractive colors etc might matter as well, just like processing load can be a factor for lossy codecs).

-k

This post has been edited by knutinh: Aug 16 2012, 13:36
Go to the top of the page
+Quote Post
greynol
post Aug 16 2012, 13:44
Post #12





Group: Super Moderator
Posts: 10345
Joined: 1-April 04
From: San Francisco
Member No.: 13167



I think it's pretty ironic that this topic was originally posted in the Listening Tests subforum.

This post has been edited by greynol: Aug 16 2012, 13:46


--------------------
Breath is found in plots and DR figures.
Go to the top of the page
+Quote Post
DonP
post Aug 16 2012, 13:53
Post #13





Group: Members (Donating)
Posts: 1477
Joined: 11-February 03
From: Vermont
Member No.: 4955



QUOTE (chrisz78 @ Aug 16 2012, 07:56) *
To me, the most reliable assessment of any signal conversion is still to subtract the output from the input, look at how large the residuals (i.e. the things lost in the processing) are in comparison to the original signal, and whether the residuals sound recognizably like the music encoded


You don't seem to understand what lossy compression is about. Note that none of the mainstream formats are designed to this metric even though it would seem the most straightforward to implement and test.

Lossywav is the only one I know that goes in this direction. You might want to look up some of the threads on that.

Go to the top of the page
+Quote Post
knutinh
post Aug 16 2012, 13:59
Post #14





Group: Members
Posts: 570
Joined: 1-November 06
Member No.: 37047



QUOTE (chrisz78 @ Aug 16 2012, 13:56) *
To me, the most reliable assessment of any signal conversion is still to subtract the output from the input, look at how large the residuals (i.e. the things lost in the processing) are in comparison to the original signal, and whether the residuals sound recognizably like the music encoded (which is bad, as in that case recognizable parts of the original recording are distorted or left out of the encoding).

Try testing a 1 millisecond delay using that method.

Then try identifying a 1 millisecond delay in a listening test.

-k
Go to the top of the page
+Quote Post
pdq
post Aug 16 2012, 15:18
Post #15





Group: Members
Posts: 3450
Joined: 1-September 05
From: SE Pennsylvania
Member No.: 24233



QUOTE (chrisz78 @ Aug 16 2012, 07:56) *
To me, the most reliable assessment of any signal conversion is still to subtract the output from the input...

That would be true if the objective were to accurately reproduce the waveform. However, what we are dealing with here is the perception of the signal.

Take as an example human color vision. The visible spectrum covers over 200 nanometers, and yet the human eye has only four kinds of light receptors. The color that we perceive is a result of the combination of wavelengths and their intensities, along with the spectral response curves of the receptors.

When you see something as a particular color, your eye can't tell which wavelengths are present at which levels. Two spectra that are very different from one another can look like the identical color. But when you subtract the two you get a very large difference between them. Does this make spectral subtraction a good way of testing color matching? Obviously not.
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 28th December 2014 - 13:49