Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Visualization of signal distortion (Read 24146 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Visualization of signal distortion

Reply #25 –
Quote from: Serge Smirnoff on
Quote from: saratoga on
I don't think thats the right interpretation of those results.  I would say that your metric is extremely sensitive to what are essentially irrelevant differences in phase/amplitude response vs. frequency, which is why the white noise test fails to give meaningful results.  The very large difference between the harmonic tests and the broadband tests suggests that actually this is not going to work very well at all with real-world signals, which are not simple harmonics.

I think that sine wave is the easiest for accurate reproduction by any device, white noise is the hardest as its waveform is the most complicated. Real-world signals are somewhere in between (histogram shows the example of such real-world signals). White noise testing is the worst-case testing for any device, all types of distortions introduced by device contribute to its degradation. I'm not sure about further interpretation of sine/white noise measurements.


I don't really see why either of those are easier or harder.  White noise has higher entropy, but that is not relevant to a DAC. 

I think the problem is that your scores diminish rapidly (and erratically) in proportion to signal bandwidth and out of proportion with how the devices would sound (for example, I doubt that the extremely low scoring white noise would sound any different whatsoever from even an ideal white noise).  The problem is as xnor described:  you are extremely sensitive to any difference, and the number of irrelevant differences between even extraordinarily high quality devices rapidly increases due to tiny differences in phase/amplitude as bandwidth increases.

Visualization of signal distortion

Reply #26 –
Is any knowledge about the human auditory system being applied to this model?  Something as simple as an equal weighting filter should help minimize irrelevant differences.  Despite what was said about audibility, I would presume this is all still supposed to be about sound quality; otherwise what's the use?

Visualization of signal distortion

Reply #27 –
I don't see how this number is about subjective sound quality. There seems to be only some very loose correlation between the two: very bad Df is more probable to actually sound bad than very good Df.

Now this would be somewhat useful if the measured devices had wildly varying Df values, but they don't.
"I hear it when I see it."

Visualization of signal distortion

Reply #28 –
If I understand correctly, the signal has a global gain correction applied based on the amplitude at low frequency (< 1 KHz).  Then a time frequency distribution is computed using ~ 400 ms windows, and then are time aligned (maybe?) to correct for some types of phase errors.  This doesn't work very well however for two reasons:

1)  You can only correct for some of the phase difference between the signals because of the finite transform window.  Things like tiny differences in the sample clock between the test device and the recording device or minute differences in the phase response of devices will not necessarily be well corrected however. 

2)  Your gain correction is global and estimated from a single point measurement. This makes you very sensitive to tiny differences in pass-band shape and flatness that are completely irrelevant, but collectively seem to dominate your broadband measurements.  For a good example of this, look at the huge difference in white noise score and low pass filtered (en50332) white noise score.  The latter is vastly different, probably because it excludes the part of the response that is closer to the device's transition band. 

One obvious way to improve this would be to measure the device's amplitude/phase response first, compute a separate score for it, and then correct for the linear part of the amplitude/phase response before performing the rest of your analysis.  This would not solve the more fundamental problem of anchoring your metric to how human hearing really works, but it would at least make your measurements resilient against tiny passband ripples and other things that are completely irrelevant in practice.

Edit:  Another suggestion.  Before you perform device measurements, perform validation measurements using noiseless synthetic test signals to assess your resilience against confounding effects.  For example, xnor's suggestion of testing filter transition band shapes and ripple.  Any metric should not depend tiny differences in passband shape for instance, so demonstrating this is the case using simple lowpass filtered white noise would be a good idea.

Visualization of signal distortion

Reply #29 –
Quote from: Serge Smirnoff on
Quote from: Arnold B. Krueger on
It is well known that difference testing is fatally flawed by the fact that every kind of noise, distortion, and other artifact (some irrelevant to fidelity) is conflated into one number or one graph.

The whole purpose of over 100 years of audio testing has been to separate things that are of interest and relevant from those that are not.

For example difference testing makes a huge point out of latency, when due to the nature of the very process of recording, latency is not of interest at all to almost all listeners.

Difference calculation which I use is not affected by latency or any other phase shifts and time stretch/shrink. In general, all isomorphic distortions of output waveform are not counted. Difference level (Df) measures only non-isomorphic differences between two signals.


OK, my example was too easy.

Doesn't change the 500 pound gorilla in the room which is that differences conflate different kinds of noises and distortions with vastly different levels of audibility into way too few numbers.

Now if you could show how you avoid conflating the effects of spectral masking or frequency dependent audibility... ;-)

Visualization of signal distortion

Reply #30 –
That would be another psychoacoustic model then, which may or may not fit real listeners' hearing, but even a simple model should certainly give more meaningful results on average regarding perceived sound quality.
"I hear it when I see it."

Visualization of signal distortion

Reply #31 –
I think it is inherently flawed to try and combine linear transfer function performance with nonlinear.  Linear can be corrected by EQ, and will anyway be connected to a pair of speakers or headphones which will have its own (typically much larger) effect.  At the same time, linear effects tend to be very large in terms of things like RMS error or difference. 

I think an RMAA-like approach is a lot better, where the linear and nonlinear components of the instrument response are isolated and presented separately.  I wouldn't use difference as a metric directly because its so hard to make work well, but if I did I would separate the linear and nonlinear components, and evaluate them independently.

Visualization of signal distortion

Reply #32 –
Ok, thank you all for feedback and sorry in advance for answering en masse.

All the remarks could be roughly divided into three groups:
  • arguments why df-metrics will not work (insufficient accounting (weighting) of psychoacoustic effects)
  • suggestions of some tests for Df parameter (mostly to show its inefficiency as PAQ metric)
  • proposals how to improve it

The second point is the easiest to address – I will publish matlab code which computes Df (with warping of output signal and producing diffrogram). Those who want to experiment with Df and various signals will have such possibility. I need a few days to prepare the code for public release (comments, usage, demo ….)

I think that polemics about drawbacks and improvements of df-metric is pure speculative now because for constructive discussion there should be example(s) of its performance. I understand that the example I've already provided is insufficient (Kohlrabi thinks “it's pure coincidence”). So my next task will be providing more tangible examples of df-metric performance. I plan to start with more detailed review of the above example and continue with other listening tests (probably I will need some help from organizers of previous HA tests as I need sound samples and versions of codecs used). This will be "live research" as I like it.

There is another reason why I think that speculative arguing about df-metrics is very difficult. Df-metrics uses radically different approach (like in case of solving equation analytically and numerically) and all those masking thresholds and frequency/phase responses are hard to translate into Df domain where waveforms and probabilities are used. This is another reason just to show how it works in real-world rather than finding pros and fighting cons.
keeping audio clear together - soundexpert.org

Visualization of signal distortion

Reply #34 –
Quote from: Serge Smirnoff on
Matlab code for computing Difference levels and building diffrograms with examples of usage - http://soundexpert.org/news/-/blogs/visual...istortion#part3



This appears to be yet another failed example of trying to gain insight into the subjective properties of listeners by means of  potentially irrelevant mathematical models.

It is very rare that you can use just a math model to prove its relevance. Most of us seem to know that the world of math is its own abstract little world.

Visualization of signal distortion

Reply #35 –
So I generated 10s of white noise sampled at 96 kHz, low pass filtered it to 22.05 kHz, that is ref.
Then I filtered this ref with a slightly different low pass filter (a bit broader transition and 0.05 dB ripple):
Df value: -25.47 dB

This figure would further degrade when considering DC blocking, aliasing, small nonlinearities ...
"I hear it when I see it."

Visualization of signal distortion

Reply #36 –
Quote from: xnor on
So I generated 10s of white noise sampled at 96 kHz, low pass filtered it to 22.05 kHz, that is ref.
Then I filtered this ref with a slightly different low pass filter (a bit broader transition and 0.05 dB ripple):
Df value: -25.47 dB

This figure would further degrade when considering DC blocking, aliasing, small nonlinearities ...

White noise is the most sensitive (in Df sense) to all kinds of processing, sine signal is the least sensitive, all other possible signals are in between. You can try for example square wave or program simulation noise (http://www.ap.com/display/file/273).

DC blocking can affect Df by introducing computational errors only which are around -150dB (assuming you experiment with 32bit signals).
keeping audio clear together - soundexpert.org

Visualization of signal distortion

Reply #37 –
Code: [Select]
x = sin(2*pi*1000*(0:96000)./96000);
y = sin(2*pi*1000*(0:96000)./96000+17.2/180*pi);


writing x into ref, y into rout, I get
Code: [Select]
df = diffrogram('ref.wav', 'rout.wav', 100, 'Mono');
...
Df values for warp frames:
-86.9714



Is it supposed to produce "just" -87 dB considering it is just a simple phase shift?


edit: It completely fails for 180°, i.e. Df is close to 0.
"I hear it when I see it."

Visualization of signal distortion

Reply #38 –
Quote from: xnor on
Code: [Select]
x = sin(2*pi*1000*(0:96000)./96000);
y = sin(2*pi*1000*(0:96000)./96000+17.2/180*pi);


writing x into ref, y into rout, I get
Code: [Select]
df = diffrogram('ref.wav', 'rout.wav', 100, 'Mono');
...
Df values for warp frames:
-86.9714



Is it supposed to produce "just" -87 dB considering it is just a simple phase shift?


edit: It completely fails for 180°, i.e. Df is close to 0.


Resulting Df values are in df vector, not in Command Window where auxiliary/debugging data is shown. The most informative is output picture of diffrogram; its name includes median, min and max values from df vector (excluding first and last values).

In the example you provided the output diffrogram [blockquote]rout.wav(96)_ref.wav(96)_mono_100-146.5356-139.1738-132.7881.png [/blockquote]gives median -139.17dB.

Also for high sample rates and substantial phase shifts the robustness of the algorithm can be increased with the parameter WarpMargin (default is 20):

Code: [Select]
df = diffrogram('ref.wav', 'rout.wav', 100, 'Mono WarpMargin:200');

which gives slightly better results:[blockquote]rout.wav(96)_ref.wav(96)_mono_100-143.8639-140.4482-137.3975.png. [/blockquote] With this parameter your 180deg phase shift is handled by the function appropriately as well: [blockquote]rout.wav(96)_ref.wav(96)_mono_100-Inf-Inf-Inf.png. [/blockquote] But 180deg is too specific case, 181deg is more indicative: [blockquote]rout.wav(96)_ref.wav(96)_mono_100-144.3608-142.1909-139.2928.png. [/blockquote] Thus, these Df values reflect computational errors not phase shifts.

keeping audio clear together - soundexpert.org

Visualization of signal distortion

Reply #39 –
Alright, I understand. I also understand that (at least given the defaults) it is very picky concerning "alignment" of the waveforms.

Here's another test:
2 sines waves, in the compared file one sine is scaled by 0.995 (-0.043 dB):

df median = -55.0298 dB
"I hear it when I see it."

Visualization of signal distortion

Reply #40 –
Quote from: xnor on
Alright, I understand. I also understand that (at least given the defaults) it is very picky concerning "alignment" of the waveforms.

Yes, there is some pickiness concerning alignment of pure sine waves; I mentioned this in "Syntax and description" inside the code. No problems with stochastic signals.

Quote from: xnor on
Here's another test:
2 sines waves, in the compared file one sine is scaled by 0.995 (-0.043 dB):

df median = -55.0298 dB

I can't reproduce the case. At least this code
Code: [Select]
x = sin(2*pi*1000*(0:96000)./96000);
audiowrite('ref.wav',x,96000,'BitsPerSample',32);
y = 0.995*sin(2*pi*1000*(0:96000)./96000+17.2/180*pi);
audiowrite('rout.wav',y,96000,'BitsPerSample',32);
df = diffrogram('ref.wav', 'rout.wav', 100, 'Mono WarpMargin:200');
results in [blockquote]rout.wav(96)_ref.wav(96)_mono_100-142.8249-140.7953-139.2521.png[/blockquote]
keeping audio clear together - soundexpert.org

Visualization of signal distortion

Reply #41 –
I meant 2 sine waves per signal, like 20 Hz and 1000 Hz:
Code: [Select]
x = sin(2*pi*1000*(0:96000)/96000)/2+sin(2*pi*20*(0:96000)/96000)/2;
y = sin(2*pi*1000*(0:96000)/96000)/2+sin(2*pi*20*(0:96000)/96000)/2*0.995;
"I hear it when I see it."

Visualization of signal distortion

Reply #43 –
You get the same Df with fref=1 sine vs fout=1 sine + some distortion product at -52 dB.

In these two cases (simple level difference for one sine, added distortion product) we get the same result from simple null difference testing (x-y):
Code: [Select]
20*log10(sqrt(sum((x-y).^2)/length(x)))



Another test:
Code: [Select]
x = sin(2*pi*1000*(0:96000)/96000)/2+sin(2*pi*20*(0:96000)/96000)/2;
y = sin(2*pi*1000*(0:96000)/96000)/2+sin(2*pi*20*(0:96000)/96000+0.2/180*pi)/2;


Df is again about -55 dB, but this this only with 0.2° phase shift for the 20 Hz sine.
"I hear it when I see it."

Visualization of signal distortion

Reply #44 –
Probably, your test cases are designed to show that Df parameter is the same in the cases which are different from human hearing perspective. Even if this is true (proofs are desirable though) it doesn't mean that Df can't be helpful in other cases. At the end, Df is a very simple instrument which measures degradation of waveforms, no more no less. As relationship between waveform and listening experience is not trivial, Df parameter is not expected to work well in all cases. I prefer to concentrate my efforts on the cases where it works. In order to show that cases I'm starting the next live research under the working title “Objective difference measurements to predict listening test results?”. The main objective of the research is to find relationship between the two using results of already completed listening tests. I'm sure that this goal can't be reached by researching numerous special cases of  particular distortions and their just-noticeable-levels for human auditory system because result of their simultaneous/integral impact on the latter (in different proportions) will still be unclear. Instead, I'm trying different approach – direct comparison between real-world audio stimuli (represented by their waveforms which can be objectively measured) and resulting listening impressions (subjectively measured). This approach seems to me more productive.

Anyway, I appreciate your testing of diffrogram function; the feedback is helpful for me as a developer.
keeping audio clear together - soundexpert.org

Visualization of signal distortion

Reply #45 –
That is exactly the problem of the method. It is akin to a visual comparison of waveforms. The futility of that exercise has been pointed out countless times on this forum.

Two slightly different looking waveforms can sound more different than two wildly different looking waveforms. Yes, I chose my examples to demonstrate that.
Another example would be noise and the same noise filtered with a 1st order allpass ... Df suffers extremely but you probably can't even hear a difference.


A Df of -25 could be the result of huge nonlinearities, but also due to some small phase shift. You'd always have to provide extensive measurements so that we could say what degraded the Df parameter...
"I hear it when I see it."

Visualization of signal distortion

Reply #46 –
I started new thread concerning already mentioned live research “Objective difference measurements to predict listening test results?” - https://www.hydrogenaud.io/forums/index.php?showtopic=110442

Diffrograms, utility and related questions are still here.
keeping audio clear together - soundexpert.org