At this point we are not trying to explain distinctions on these diffrograms and their impact on perceived audio quality of the devices. We can just establish the fact that various signals are transferred through these devices with various precision.

I would like to propose a method of visualization of differences between two signals – input and output. Both signals are divided into time blocks (50+ ms) and difference between each corresponding blocks is calculated in dB and indicated by color. Such representation of difference signal helps to spot visually what parts of signal are transferred less accurately through device or processing under test. For example, it easily shows that sound section of iPhone 6 is exactly the same as of iPhone 5S.

Example of diffrogram for iPhone 6 and nine SE sound samples is below.



Details of the method and diffrograms for other devices are in the article “Diffrogram: visualization of signal differences in audio research”. Your critique and opinions would be very helpful for me. Thanks in advance.

Even better, a plot of SNR vs. frequency as a function of time. Then you have a chance of making minimal sense of it.

Part 2 of the article is ready - http://soundexpert.org/news/-/blogs/visual...istortion#part2
Testing of iBasso DX50 with standard test signals and real-life music material.

p.s. with color pictures!!!

Starting with Figure 21, things that are clear to me in other presentation formats start looking like mud to me.  I can get spectrograms and obtain useful knowledge from them, but they are not as good for me other formats for the same information.

Example of testing digital audio player with various audio signals and single measurement approach based on Difference level parameter.

Quote from: Arnold B. Krueger on 2015-01-29 13:57:26

Starting with Figure 21, things that are clear to me in other presentation formats start looking like mud to me.  I can get spectrograms and obtain useful knowledge from them, but they are not as good for me other formats for the same information.

Spectrograms are supplementary here, they are just to better understand the nature of the signal being researched. The main idea of diffrogram is to present a map of output waveform degradation (and to measure it). So the useful knowledge is what parts of a signal (or what types of signal) are distorted the most (or the less) by some device under test.

It is well known that difference testing is fatally flawed by the fact that every kind of noise, distortion, and other artifact (some irrelevant to fidelity) is conflated into one number or one graph.

The whole purpose of over 100 years of audio testing has been to separate things that are of interest and relevant from those that are not.

For example difference testing makes a huge point out of latency, when due to the nature of the very process of recording, latency is not of interest at all to almost all listeners.

Finally new measurement procedure and presentation of results were defined. I started collection of audio measurements for portable players here – http://soundexpert.org/portable-players. Df-slides are self-explanatory; top ones represent devices supposed to have better perceived quality (more transparent).

Wow, this is awesome. I'd love to see how the Pono stacks up to these other devices.

Df [dB] = 10 * Log10(1 - |p|)

So in

Code: [Select]

Df [dB] = 10 * Log10(1 - |p|)

is p simply the cross-correlation of the reference and recorded signal?

I don't see how that would correspond well to subjective evaluation at all. Small audibly irrelevant differences would degrade Df strongly.

Quote from: xnor on 2015-09-15 22:14:07

So in

Code: [Select]

Df [dB] = 10 * Log10(1 - |p|)

is p simply the cross-correlation of the reference and recorded signal?

I don't see how that would correspond well to subjective evaluation at all. Small audibly irrelevant differences would degrade Df strongly.

Df is the cross-correlation but not simple. It is calculated only after output signal is time and phase aligned with desired accuracy. So "irrelevant differences" are not counted at all.

How do you account for irrelevent differences in the amplitude or phase response of the device's output ?

I am not talking about phase but e.g. ripple in low pass filters, the effects of low frequency roll-off etc.

Looking at some of those results where you have near 16 bit limited performance for pure tones but single digit dB for white noise, it looks like you are not.

Quote from: saratoga on 2015-09-15 22:36:32

Looking at some of those results where you have near 16 bit limited performance for pure tones but single digit dB for white noise, it looks like you are not.

Right you are, reproduction of white noise is heavily affected by phase response. As you can see reproduction of real-life sound material is affected to substantially lesser degree (exactly as you want it to be).

Then those medians were mapped to results of the listening test

Taking into account that different sound material was used for calculation of Df and for listening test the correlation is surprising, blue dots are almost inside confidence intervals of audio quality scores.

Is it just a coincidence or a sign of some regularity that was left unnoticed during “over 100 years of audio testing ”?

Today I'm pretty sure that difference testing has a huge research potential, not as a substitute but  a supplement to listening tests. When nature of waveform degradation is the same, Df correlates well with results of listening tests. In our case Opus has substantially different psy-model (which is also seen in its histogram) and Df magic does not work for it.

So, difference testing is not “fatally flawed” it was just not used correctly.

Quote from: Serge Smirnoff on 2015-08-15 11:50:10

Then those medians were mapped to results of the listening test

How?

OK, so your metric fails completely to correlate with the perceived quality of this test, since only a single one of your mapped red dots fits into the confidence interval of one of the results, and even that single point might only fit due to how you do your mapping. Then you artificially removed Opus from your data, which, in fact, is the most interesting data point of all, because the Df metric is so vastly different from the perception based result. This shifts your dots around, and all of the sudden more dots fit the coincidence intervals. So...

It's pure coincidence, and it's also not a very good correlation at all. You provided no rationale why your Df metric should correlate at all with the result of Kamedo's test. All I see is random cherry-picking and doctoring of the data. Again.

• sometimes “ cherry-picking and doctoring of the data” means research
• you don't need “a million” cases to prove something, a few good ones are usually enough
• all scientific knowledge is falsifiable by definition

I don't think thats the right interpretation of those results.  I would say that your metric is extremely sensitive to what are essentially irrelevant differences in phase/amplitude response vs. frequency, which is why the white noise test fails to give meaningful results.  The very large difference between the harmonic tests and the broadband tests suggests that actually this is not going to work very well at all with real-world signals, which are not simple harmonics.

I think that sine wave is the easiest for accurate reproduction by any device, white noise is the hardest as its waveform is the most complicated. Real-world signals are somewhere in between (histogram shows the example of such real-world signals). White noise testing is the worst-case testing for any device, all types of distortions introduced by device contribute to its degradation. I'm not sure about further interpretation of sine/white noise measurements.