IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
AES conference London: High Resolution perception, paper about listening test
Kees de Visser
post Jul 5 2007, 11:25
Post #1





Group: Members
Posts: 649
Joined: 22-May 05
From: France
Member No.: 22220



Unfortunately I wasn't able to visit the june 2007 AES Conference in London about High Resolution Audio.
The paper/presentation about a high-res audio listening test seems interesting. I'm wondering if anyone on HA happens to have been there and can share some information.

preview of the paper session:
QUOTE
Monday, June 25 11:00 – 12:30
Paper Session 2 — Perception

2-1 Which of the Two Digital Audio Systems Meets Best with the Analog System?— Wieslaw Woszczyk,1 Jan Engel,2 John Usher,1 Ronald Aarts,3 Derk Reefman3
1McGill University, Montreal, Quebec, Canada
2Centre for Quantitative Methods CQM BV
3Philips Research, Eindhoven, The Netherlands

In this listening test, two digital audio systems (B and C), and one analog system (A) were tested by 10 test persons who listened to a surround sound scene “live” (without recording). The main question to be answered was: “Which of the two digital systems meets best with the analog system?” Both digital versions had 24-bit dynamic resolution but differed in sampling rate with which the analog signal was sampled. One version © was sampled with a CD rate of 44.1 kHz, the other (B) 8 times faster. There were also two test conditions, where in one condition there was a bandwidth cut off at 20 kHz instead of the 100 kHz that was possible with special 100 kHz microphones and added super-tweeters. For each subject, the experiment was replicated six times, in each of the two conditions. The outcome of each experiment was a 0 or 1, where the 1 means that the, technically best, digital system B has been chosen as meeting the analog quality. The paper describes the test and the outcome.

Without having read the paper, it's not clear to me whether the test was double-blind or not. Apparently it was not possible to replay sources, since the audio source was "live". How reliable would a test like this be ?
Go to the top of the page
+Quote Post
krabapple
post Jul 5 2007, 16:40
Post #2





Group: Members
Posts: 2184
Joined: 18-December 03
Member No.: 10538



In any case it appears to be a test to see which of two digital systems sounds most like an analog one, which is considered 'technically best'. I'd be curious to see the reasoning behind such a claim. Are papers being presented, peer-reviewed?

This post has been edited by krabapple: Jul 5 2007, 16:41
Go to the top of the page
+Quote Post
krabapple
post Dec 10 2007, 08:41
Post #3





Group: Members
Posts: 2184
Joined: 18-December 03
Member No.: 10538



QUOTE (krabapple @ Jul 5 2007, 10:40) *
In any case it appears to be a test to see which of two digital systems sounds most like an analog one, which is considered 'technically best'. I'd be curious to see the reasoning behind such a claim. Are papers being presented, peer-reviewed?


So far though I haven't been able to find any details of the presentation. From the abstract , if nothing else it looks to me like M&M's sample pool was much deeper....

FWIW, John Atkinson is touting this presentation as evidence against the claims of Meyer and Moran's paper, in the current Stereophile (not online yet)

Here's what he wrote about it in September:

http://stereophile.com/asweseeit/907awsi/



QUOTE
I will end this month's essay by quoting, from a paper given at the conference, the results of experiments on the audibility of high sampling rates: "To achieve a higher degree of fidelity to the live analog reference, we need to convert audio using a high sampling rate even when we do not use microphones and loudspeakers having bandwidth extended far beyond 20kHz. Listeners judge high sampling conversion as sounding more like the analog reference when listening to standard audio bandwidth." (footnote 2)

So that's that, then.


And a month alter he reports a blind test he participated in with various sample rates and mp3 bitrates

http://stereophile.com/asweseeit/1007awsi/

I suspect he's full of shit, frankly, at least as far as getting the whole story, is concerned.

This post has been edited by krabapple: Dec 10 2007, 08:51
Go to the top of the page
+Quote Post
Kees de Visser
post Dec 10 2007, 09:23
Post #4





Group: Members
Posts: 649
Joined: 22-May 05
From: France
Member No.: 22220



QUOTE (krabapple @ Dec 10 2007, 08:41) *
So far though I haven't been able to find any details of the presentation. From the abstract , if nothing else it looks to me like M&M's sample pool was much deeper....
Finally the paper is online as pdf here. I've had a copy for quite some time but discussion is difficult until a paper becomes publicly available.
Enjoy. It's one of those tests the high-resolution proponents have been waiting for.

Kees de Visser

This post has been edited by Kees de Visser: Dec 10 2007, 11:04
Go to the top of the page
+Quote Post
cabbagerat
post Dec 10 2007, 10:50
Post #5





Group: Members
Posts: 1018
Joined: 27-September 03
From: Cape Town
Member No.: 9042



This paper is certainly worth reading for anybody interested in the subject.

QUOTE
There were also two test conditions, where in one condition there was a bandwidth cut off at 20 kHz instead of the 100 kHz that was possible with special 100kHz microphones and added super-tweeters.
This is extremely interesting. The authors write about tests done at NHK to identify the audibility of ultrasonics (above 21kHz), and seem to have chosen to concentrate only on the low frequencies:
QUOTE
(1) Is there a possible benefit of high-resolution audio at lower frequencies without the necessity for the reproduction of supersonic components?
Their second goal is even more interesting:
QUOTE
Is a noticeable benefit of high-resolution revealed in surround sound listening?


The results are extremely interesting. Some extracts:
QUOTE
This means that at the 100 kHz bandwidth the low sampling system Y is more often than X judged to be most like the analog system A.

QUOTE
This means that with the cut-off at 20 kHz bandwidth the high sampling system X is more often than Y judged to be most like the analog system A.


In the conclusions, they hypothesize that the wide bandwidth system sounds less transparent because of what they call "noise-like artifacts" in the high frequencies (above 20kHz), which are not passed in the lower sampling rate version of the wide bandwidth system. I am not sure how they support this hypothesis, given that the analog reference system described also passed the ultrasonics.

They also claim that "Listeners judge high sampling conversion as sounding more like the analog reference when listening to standard audio bandwidth." If this effect can be reproduced in other papers using similar equipment, it suggests that the practical limitations in sampled data systems need to be addressed through oversampling (that is, sampling significantly above the Nyquist limit).

This post has been edited by cabbagerat: Dec 11 2007, 10:32


--------------------
Simulate your radar: http://www.brooker.co.za/fers/
Go to the top of the page
+Quote Post
Kees de Visser
post Dec 10 2007, 11:36
Post #6





Group: Members
Posts: 649
Joined: 22-May 05
From: France
Member No.: 22220



QUOTE (cabbagerat @ Dec 10 2007, 10:50) *
In the conclusions, they hypothesize that the wide bandwidth system sounds less transparent because of what they call "noise-like artifacts" in the high frequencies (above 20kHz), which are not passed in the lower sampling rate version of the wide bandwidth system. I am not sure how they support this hypothesis, given that the analog reference system described also passed the ultrasonics.
It's not very clear in the paper, but IMO it's possible that ultrasonic artifacts are introduced, added and/or modified in the AD/DA process.
It's also important to know that only two sample rates were used: 44.1 and 352.8 kHz, derived from a 128xFs 5 bit delta-sigma modulator. It would have been interesting to know if popular standards like 96 and 192 kHz would behave differently. I can imagine however that time and budget were limited.
Note also that only one brand of AD/DA converter has been used (Digital Audio Denmark). AFAIK it hasn't been tested if results remain identical with other brands.

Kees de Visser

This post has been edited by Kees de Visser: Dec 10 2007, 11:38
Go to the top of the page
+Quote Post
krabapple
post Dec 11 2007, 08:21
Post #7





Group: Members
Posts: 2184
Joined: 18-December 03
Member No.: 10538



So , how to reconcile these with Meyer and Moran's trials, which used several SACD players, and more subjects, with the conclusion that CD and SACD sound the same at normal volumes? Does it come down to matching digital formats against analog, versus against each other? Surround instead of two channel? THe use of a highly contrived 'musical scene' (an electric motor driving a bicylce wheel in an anaechoic chamber??!!) as probe signal, versus recorded music? THat some listeners may have been in fact judging ANALOG (the live feed) versus DIGITAL (both A/D conversions), in a nonblind comparison?

This post has been edited by krabapple: Dec 11 2007, 08:33
Go to the top of the page
+Quote Post
cabbagerat
post Dec 11 2007, 08:48
Post #8





Group: Members
Posts: 1018
Joined: 27-September 03
From: Cape Town
Member No.: 9042



QUOTE (Kees de Visser @ Dec 10 2007, 02:36) *
It's also important to know that only two sample rates were used: 44.1 and 352.8 kHz, derived from a 128xFs 5 bit delta-sigma modulator. It would have been interesting to know if popular standards like 96 and 192 kHz would behave differently. I can imagine however that time and budget were limited.

I found the choice of 352.8 kHz fairly interesting, but I suppose that deriving the signal from the same ADC with the same clock was the only way to keep the test fair. The paper is very light on the details of the design of the digital chain, which leads me to wonder what impact it could have had on the test results. DACs are known to change in performance (notably THD+N) with different sample rates (see the datasheets of most sigma-delta audio DACs for reference).
QUOTE (Kees de Visser @ Dec 10 2007, 02:36) *
Note also that only one brand of AD/DA converter has been used (Digital Audio Denmark). AFAIK it hasn't been tested if results remain identical with other brands.
Absolutely. The fact that the paper doesn't make strong conclusions about audibility, however, means that it's less important to reproduce the results of the trials to support the conclusions.


--------------------
Simulate your radar: http://www.brooker.co.za/fers/
Go to the top of the page
+Quote Post
MichaelW
post Dec 11 2007, 09:59
Post #9





Group: Members
Posts: 631
Joined: 15-March 07
Member No.: 41501



One thing I see is that although the criterion is which digital system is more like the analogue, the discussion slips into using words like "preferred." Don't know if that makes a difference.

They also seem to be making heavy weather of the fact that their subjects found it hard to tell the difference -- they suggest redesigning the experiment, rather than accepting that the differences are small.

But it's interesting.
Go to the top of the page
+Quote Post
Pio2001
post Dec 11 2007, 16:45
Post #10


Moderator


Group: Super Moderator
Posts: 3936
Joined: 29-September 01
Member No.: 73



Thank you for the link, Kees de Visser !

All I can see is that the probabilities that the listeners heard differences are 40 % and 60 %.

This is inferior to the minimum 95 % of statistic relevance. Thus in both conditions, listeners failed to demonstrate that any difference was audible between the tested stimuli.

I don't understand the point in discussing the fact that 40 % can be said to be inferior to 60 % with 95 % certainty, since both are below the required threshold for statistical significance anyway.

This post has been edited by Pio2001: Dec 11 2007, 16:57
Go to the top of the page
+Quote Post
SoleBastard
post Dec 24 2007, 10:38
Post #11





Group: Members
Posts: 65
Joined: 13-August 04
Member No.: 16237



Man, this article has the worst abstract I have ever seen! I had to plough through the whole rag-tag thing just to figure out what exactly they had done! Still not sure if I understand it though...

As I can see it, its even worse than Pio2001 states: The 95% CI intervals between C1 and C2 overlap ironically @ 0,5, which make them equal. Thus no conclusions can be drawn (as stated at conclusion 4). I won't even comment on conclusion 5 laugh.gif.

The whole thing about the lowpassed high resolution version being perceived as better than the non-lowpassed high resolution is very interesting. Their hypothesis that ultrasound frequencies actually cause more artefacts seems to go against their recommendation of using a non-lowpassed high resolution for archiving!

Some more interesting bits of information from the article:

- This appreciation must relate to a number of perceived characteristics other than high-frequency response partly because microphones and loudspeakers normally used in studios do not have a substantially wider response than the audible range.

- In double-blind tests, casual and professional listeners could not reliably identify high-bandwidth and high-resolution (192 kHz 24bit versus 48 kHz 24bit) conditions.
Go to the top of the page
+Quote Post
boombaard
post Dec 24 2007, 12:02
Post #12





Group: Members
Posts: 336
Joined: 7-February 05
From: Local Cluster
Member No.: 19647



must've missed it when you posted it first.. odd.

anyway

QUOTE
However, to achieve a higher degree of fidelity to the live analog reference, we need to convert audio using high sampling rate even when we do not use microphones and loudspeakers having bandwidth extended far beyond 20 kHz. Listeners judge high sampling conversion as sounding more like the analog reference when listening to standard audio bandwidth.


and this quote struck me as very, very odd.
(read the article first, or at least the conclusions/test setup bits before posting.. the phrasing seems to be a bit awkward in places, but i can't really help that)

because unless i'm really, really reading this wrong they're saying that if you upsample to the highest SR your sound card can take before playing it back - no matter the quality of the input material - it will be perceived as 'more natural'.

(mind you, i don't really see why you'd need to do what they suggest next, namely:
QUOTE
These results suggest that the archiving community should consider using high-sampling conversion to ensure transparency even if the recording is made with standard audio-bandwidth transducers, and when digitizing older recordings made with bandwidth-limited analog systems.
just turning on SSRC in your foobar dsp chain would seem to be enough)

the point, really, is that i almost can't believe that i'm understanding this right.
A second point: even if they are correct, it would seem that their conclusion that anything >20kHz is a waste of bw is somewhat hasty, and they imo should've added a 'middle' test - to say 40kHz - just to narrow down where the added bandwidth became overkill (more).

QUOTE (Kees de Visser @ Dec 24 2007, 09:04) *
PS:
QUOTE
because unless i'm really, really reading this wrong they're saying that if you upsample to the highest SR your sound card can take before playing it back - no matter the quality of the input material - it will be perceived as 'more natural'.
No, it's not about upsampling. The test compares an AD/DA chain to the analog source at two differents sample rates: 44.1 and 352.8 kHz. This is basically different from upsampling 44.1 kHz data that already have passed a low pass filter.


well, the topic of the paper is at least remarkably on-topic.. but i don't really see what you base your statement about the AD/DA chain on.
as i understand it, both the 20kHz recording equip and the 100kHz ones were attached to the same chain, so that doesn't really seem relevant here.
furthermore:
QUOTE
In the 20 kHz case (C2), they liked the high-sampling version, which would have an effect on other aspects of technical quality than the high frequency response. This may indicate that we do not need 100 kHz microphones and 100 kHz loudspeakers, but we do need 100 kHz capable recorders.


so what are they saying here then? that 20kHz cutoff mikes still produce overtones/noise in the 20-100kHz band that's relevant to the recording and is perceived as adding to the perceived 'naturalness' of said recording, even though they're not 'real'?

anyway, what i don't really see is why that would influence the archiving community.. i can see how this might be interesting to recording studios, but by the time it reaches the consumer, the format is already decided on. so why would it be relevant to him to save it in as high a sampling rate as possible?
Go to the top of the page
+Quote Post
Kees de Visser
post Dec 24 2007, 15:32
Post #13





Group: Members
Posts: 649
Joined: 22-May 05
From: France
Member No.: 22220



QUOTE (boombaard @ Dec 24 2007, 12:02) *
anyway, what i don't really see is why that would influence the archiving community.. i can see how this might be interesting to recording studios, but by the time it reaches the consumer, the format is already decided on. so why would it be relevant to him to save it in as high a sampling rate as possible?
Aha, perhaps I've found the source of confusion. AFAIK the paper assumes archiving to be the process of preventing (historical) analog source material from deterioration by transferring it to a new medium, which in these days usually means a digital format. On the first page of the paper:
QUOTE
Introduction:
The audio archiving community responsible for the preservation of our sonic cultural heritage is interested in adopting a digital conversion and storage format that can be considered transparent by listeners skilled in the art of audio. Therefore, a digital medium having high degree of fidelity to the analog reference is needed.
Analog tapes don't age very well and at some day it will be impossible to play them back at all. If it is important to preserve the audio quality of the original, an as transparent as possible conversion process should be chosen. When the analog originals have become useless, all that's left is the copy, so the choice of archive format is very important. The paper gives some evidence that high sample-rate formats provide a copy that is sonically closer to the original compared to standard rate formats.
Since the cost of a higher rate format in the whole process of archiving is rather small, this could be important information for professional archiving institutions.
Go to the top of the page
+Quote Post
Pio2001
post Dec 25 2007, 23:58
Post #14


Moderator


Group: Super Moderator
Posts: 3936
Joined: 29-September 01
Member No.: 73



QUOTE (SoleBastard @ Dec 24 2007, 10:38) *
The whole thing about the lowpassed high resolution version being perceived as better than the non-lowpassed high resolution is very interesting.


QUOTE (boombaard @ Dec 24 2007, 12:02) *
they're saying that if you upsample to the highest SR your sound card can take before playing it back - no matter the quality of the input material - it will be perceived as 'more natural'.


Until a double blind test can show that high resolution is sonically different than standard resolution, any listener can state anything according to his own taste.
All DBT that have succeeded were flawed one way or another.

This one is not flawed, it just failed.

QUOTE (Kees de Visser @ Dec 24 2007, 15:32) *
The paper gives some evidence that high sample-rate formats provide a copy that is sonically closer to the original compared to standard rate formats.


It doesn't. The results are far below statistical confidence. The test failed to show any difference between the stimuli.

The p values are given in figure 1.
They are 0.4 and 0.6, while they should have been at most 0.01 or at least 0.99 for a type I error probability of 5 % (probability to have got the result by chance) because there are four possibilities of success : p1 <0.01 OR p1 > 0.99 OR p2 < 0.01 OR p2 > 0.99, which roughly multiplies the probability of getting a success by 4.

This post has been edited by Pio2001: Dec 26 2007, 00:00
Go to the top of the page
+Quote Post
tarsier
post Jan 1 2008, 06:35
Post #15





Group: Members
Posts: 22
Joined: 26-December 05
Member No.: 26654



QUOTE (Pio2001 @ Dec 25 2007, 15:58) *
This one is not flawed, it just failed.

I'm having a hard time sorting through the math, someone tell me if I'm offbase...

They wanted to test whether sample rates of 44.1 kHz or 8 x 44.1 kHz sounded more like the original source. For those two choices there were two conditions tested: C1 (100 kHz audio bandwidth) and C2 (20 kHz audio bandwidth). (p.3, very last paragraph).

Under C1 conditions, the probability that the high sample rate (8 x 44.1 kHz) sounded more like the analog was 0.4 (p.8 table 3). Does that mean that the probability that the low sample rate (44.1 kHz) sounded more like the analog was 0.6?

And then for C2 (table 3 again), 0.6 probability that high sample rate sounded more like the analog. So does that mean 0.4 probability that low sample rate sounded more like analog?

Am I reading the numbers correctly?

If so, then I have to agree with Pio2001. This test failed to show that either of them sounded "more" like the analog. There was no statistical significance either way. I don't understand how conclusions 2 and 3 on p.9 can say "significantly smaller" and "significantly higher" when it seems like they really should say "a bit smaller" and "a bit higher".
Go to the top of the page
+Quote Post
Kees de Visser
post Mar 10 2012, 10:30
Post #16





Group: Members
Posts: 649
Joined: 22-May 05
From: France
Member No.: 22220



The OP pdf document (2007) has been moved:
http://www.extra.research.philips.com/hera...rs/aar07pu4.pdf

I noticed some extra comments (2009) from the authors:
http://www.extra.research.philips.com/hera...s/aar09pu13.pdf

Kees de Visser
Go to the top of the page
+Quote Post
krabapple
post Mar 10 2012, 22:49
Post #17





Group: Members
Posts: 2184
Joined: 18-December 03
Member No.: 10538



Does anyone know who the 'reader' is , whose comments are being replied to by the authors?

Anyway, from those replies this struck me as odd
QUOTE
Assumption 2: “Both digital systems X and Y have a
performance that is not better than the performance of the
analog system A, if we measure this performance on the
one-dimensional latent variable.”
Assumption 2 has reasonable logic from audio knowledge
and experience if we assume that in theory the analog
version has an infinite number of data points (that is,
infinite resolution
), and DXD (8 Fs) has used eight times
more data points for conversion than the 1-Fs system.


emphasis mine

This 'theoretical' analog version is, therefore, a perfect recording. But real analog never is, and it never has 'infinite resolution', so why assume that digital is not better?

and this
QUOTE
Considering that the experience of professional recording
and mastering engineers working with high-resolution
audio has not been confirmed and quantified in laboratory
tests, we do not have a good validation of the testing
methods used in our industry. The current subjective testing
methodology does not seem to sufficiently reveal or
amplify the features characterizing individual listening
experiences. Perhaps this methodology, which is derived
from food and fragrance testing, is not as readily effective
for investigating subtle auditory sensations and the experience
of music? We too would like to encourage more
work in this area.


seems to me just Stereophiliac bafflegab. DBTs persistently fail to validate the experiences of 'professional recording and mastering engineers' who routinely employ only inherently flawed sighted comparison methods...therefore the problem may be with DBT? Come on!

This post has been edited by krabapple: Mar 10 2012, 22:55
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 31st July 2014 - 08:49