IPB

Welcome Guest ( Log In | Register )

3 Pages V  < 1 2 3 >  
Reply to this topicStart new topic
24/96 digitalization - can it be audible, blind test results
WernerO
post Jun 21 2011, 11:26
Post #26





Group: Members
Posts: 79
Joined: 21-November 06
Member No.: 37858



QUOTE (AndyH-ha @ Jun 21 2011, 10:01) *
If so, where can some revealing results be found?


PCM1804 datasheet.



Not what I would call a significant problem.

Go to the top of the page
+Quote Post
Northpack
post Jun 21 2011, 15:53
Post #27





Group: Members
Posts: 457
Joined: 16-December 01
Member No.: 664



I don't think that we have to speculate about possible minor shortcomings of the Tascam when the test is known to have two major problems:

- Level matching within 0.2dB is insufficient and could be well audible
- Clipping

It should be repeated with levels matched within 0.1dB and lowered input level to give really significant results.
Go to the top of the page
+Quote Post
Martel
post Jun 21 2011, 17:26
Post #28





Group: Members
Posts: 568
Joined: 31-May 04
From: Czech Rep.
Member No.: 14430



I'm no statistics expert but it seems like too little data. Have you tried flipping a coin 130 times instead and processing the results the same way? What kind of conclusion would you jump to if it gave you the same result? biggrin.gif


--------------------
IE4 Rockbox Clip+ AAC@192; HD 668B/HD 518 Xonar DX FB2k FLAC;
Go to the top of the page
+Quote Post
pawelq
post Jun 21 2011, 19:26
Post #29





Group: Members
Posts: 541
Joined: 20-December 05
From: Springfield, VA
Member No.: 26522



QUOTE (krabapple @ Jun 19 2011, 22:55) *
Just to be clear, this is all about an average score? All the individual scores failed to reach the statistical significance threshold?
Yes - but keep in mind that due to low number of trials and due to correction for multiple participants, the threshold for individual analysis was very high - 12/13 =92.3%. Unfortunately we could not run more trials, with 50 trials and 10 participants just 74% (37/50) would suffice.

QUOTE (2Bdecided @ Jun 20 2011, 05:11) *
82/130.

Assuming you didn't cherry pick the data, I reckon it's fair to take that as one block.
I did not cherrypick. All data from all participants are used, exactly as reported in the asnwer cards. Actually, one of the top-scorer's errors was a correction from initially correct response, had he not corrected, he would pass the individual statistical threshold. But we take his (wrong) correction as final.

I refrained from taking the data as one block as you suggested, because they were 13 submeasurents (trials) in 10 measurements (participants), and assuming individual differences in perceptual abilities, the 13 submeasurements within each of 10 participants were not independent. Anyway, the 95% confidence interval of the 82/130 proportion is 0.5452 to 0.7089 (or 0.5413 to 0.7125, using another method), it does not include 0.5, so would I consider the proportion to be significantly different from 0.5.

QUOTE (2Bdecided @ Jun 20 2011, 05:11) *
I'd be very worried about a 0.2dB level difference, and quite worried about clipping (depending on the content).
Yeah, clipping is not good, and we should have avoided it. As i said though, there was very little of it, just a few cut of peaks over tens of seconds of recording.

As of the level difference, are there any published data showing sensitivity to level differences <0.2 dB in complex/musical signals? Anyway, if this is significant, we would need a different A/D/A device, one that it is closer to ideal unity gain

QUOTE (usernaim @ Jun 20 2011, 11:21) *
But the multiple listeners issue is bothersome. Is there any possibility that the listeners influenced each other?
This cannot be excluded, but there was no way to get access to the room and equipment for 10 individual sessions. We have no reports of such influence from the participants, but subconscious influence is theoretically possible

QUOTE (usernaim @ Jun 20 2011, 11:21) *
Also, as far as applicability, we don't tend to listen to our hi-fi that way, with all that acoustic interference.
What kind of acoustic interference are you referring to?

QUOTE (Northpack @ Jun 21 2011, 10:53) *
- Level matching within 0.2dB is insufficient and could be well audible
[...]
It should be repeated with levels matched within 0.1dB and lowered input level to give really significant results.
Again, I'd appreciate references that established jnd for intensity in complex signals to be <0.2 dB

QUOTE (Martel @ Jun 21 2011, 12:26) *
I'm no statistics expert but it seems like too little data. Have you tried flipping a coin 130 times instead and processing the results the same way? What kind of conclusion would you jump to if it gave you the same result? biggrin.gif
I believe this part has been taken care of properly without physically flipping a coing. That's what we have math for.

This post has been edited by pawelq: Jun 21 2011, 19:27


--------------------
Ceterum censeo, there should be an "%is_stop_after_current%".
Go to the top of the page
+Quote Post
2Bdecided
post Jun 22 2011, 10:23
Post #30


ReplayGain developer


Group: Developer
Posts: 5364
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



QUOTE (pawelq @ Jun 21 2011, 19:26) *
QUOTE (2Bdecided @ Jun 20 2011, 05:11) *
I'd be very worried about a 0.2dB level difference, and quite worried about clipping (depending on the content).
Yeah, clipping is not good, and we should have avoided it. As i said though, there was very little of it, just a few cut of peaks over tens of seconds of recording.

As of the level difference, are there any published data showing sensitivity to level differences <0.2 dB in complex/musical signals?

QUOTE (Northpack @ Jun 21 2011, 10:53) *
- Level matching within 0.2dB is insufficient and could be well audible
[...]
It should be repeated with levels matched within 0.1dB and lowered input level to give really significant results.
Again, I'd appreciate references that established jnd for intensity in complex signals to be <0.2 dB
There are none*. But then, there are no published references showing that a 20kHz LPF and 16-bit quantisation is audible in musical signals either. So, great care must be taken.

* - The closest is 0.25dB under quite different conditions: F.E. Toole and S. Olive, "The Modification of Timbre by Resonances: Perception and Measurements", JAES vol 36, # 3, March 1988, pp 122-142

Cheers,
David.
Go to the top of the page
+Quote Post
Martel
post Jun 22 2011, 13:43
Post #31





Group: Members
Posts: 568
Joined: 31-May 04
From: Czech Rep.
Member No.: 14430



QUOTE (pawelq @ Jun 21 2011, 20:26) *
I believe this part has been taken care of properly without physically flipping a coing. That's what we have math for.

Well, any statistical analysis is easily misinterpreted or skewed towards a certain point by formulating the tested hypothesis in a certain way.
Why not test a hypothesis that all of the participants were just guessing and their scores were drawn from a coin-flip-distribution random number "generator"?
That was my point, sorry for not making it more apparent. smile.gif


--------------------
IE4 Rockbox Clip+ AAC@192; HD 668B/HD 518 Xonar DX FB2k FLAC;
Go to the top of the page
+Quote Post
pawelq
post Jun 22 2011, 13:55
Post #32





Group: Members
Posts: 541
Joined: 20-December 05
From: Springfield, VA
Member No.: 26522



QUOTE (2Bdecided @ Jun 22 2011, 05:23) *
There are none*. But then, there are no published references showing that a 20kHz LPF and 16-bit quantisation is audible in musical signals either. So, great care must be taken.


Sure. I am trying to find out how close level-matching needs to be. In other words, what is the basis and the limit for the "better matching" requests. I mmagined that we would do a similar test with levels matched to 0.06 dB, and how do I know that a HA'er won't say "hey, it should have been 0.05 or better".


QUOTE (Martel @ Jun 22 2011, 08:43) *
Why not test a hypothesis that all of the participants were just guessing and their scores were drawn from a coin-flip-distribution random number "generator"?
That was my point, sorry for not making it more apparent. smile.gif


I suggest that you re-read the thread, paying special attention to "binomial test", "Šidák correction", and "confidence interval of proportion".



--------------------
Ceterum censeo, there should be an "%is_stop_after_current%".
Go to the top of the page
+Quote Post
Northpack
post Jun 22 2011, 14:39
Post #33





Group: Members
Posts: 457
Joined: 16-December 01
Member No.: 664



QUOTE (pawelq @ Jun 22 2011, 12:55) *
QUOTE (2Bdecided @ Jun 22 2011, 05:23) *
There are none*. But then, there are no published references showing that a 20kHz LPF and 16-bit quantisation is audible in musical signals either. So, great care must be taken.

Sure. I am trying to find out how close level-matching needs to be. In other words, what is the basis and the limit for the "better matching" requests. I mmagined that we would do a similar test with levels matched to 0.06 dB, and how do I know that a HA'er won't say "hey, it should have been 0.05 or better".

AFAIK the common standard for level-matching is <= 0.1dB. I don't know whether this value has been empirically deducted but it seems to be sufficient.

This post has been edited by Northpack: Jun 22 2011, 14:40
Go to the top of the page
+Quote Post
Axon
post Jun 22 2011, 16:24
Post #34





Group: Members (Donating)
Posts: 1985
Joined: 4-January 04
From: Austin, TX
Member No.: 10933



FWIW, I recall once being able to successfully ABX a synthetic signal to at least a 0.05db difference in volume, with IEMs. But I did not think I could have been able to repeat that feat with speakers or with "real" music, and I've always worried that those Etys might have distortion levels varying significantly with sound level....

This post has been edited by Axon: Jun 22 2011, 16:25
Go to the top of the page
+Quote Post
knutinh
post Jun 23 2011, 12:29
Post #35





Group: Members
Posts: 570
Joined: 1-November 06
Member No.: 37047



QUOTE (pawelq @ Jun 22 2011, 14:55) *
QUOTE (2Bdecided @ Jun 22 2011, 05:23) *
There are none*. But then, there are no published references showing that a 20kHz LPF and 16-bit quantisation is audible in musical signals either. So, great care must be taken.


Sure. I am trying to find out how close level-matching needs to be. In other words, what is the basis and the limit for the "better matching" requests. I mmagined that we would do a similar test with levels matched to 0.06 dB, and how do I know that a HA'er won't say "hey, it should have been 0.05 or better".

Good point.

I guess one answer woul be that "the more controversial your findings are, and the less able others are to repeat your test, the more painstaking you have to be in controlling these things in order to make a difference to 95% of the sane readers. The last 5% will dispute your findings no matter what you do".

0.1dB is a number that I have seen several times. It might be a number that is both practical to achieve and well enough ahead of what we believe that humans can distinguish.

I think that at this point, getting independent confirmation is most important.

-k
Go to the top of the page
+Quote Post
Brand
post Jun 23 2011, 13:38
Post #36





Group: Members
Posts: 321
Joined: 27-November 09
Member No.: 75355



FWIW, I just ABXed a song with a 0.1 dB difference (normalized the original, which was at 0.0 to -0.1). I used headphones and focused on a very short sample repeatedly. So probably this is not the same methodology as you used, but it's still something I'd look into for such tests.

This post has been edited by Brand: Jun 23 2011, 13:40
Go to the top of the page
+Quote Post
Northpack
post Jun 23 2011, 14:34
Post #37





Group: Members
Posts: 457
Joined: 16-December 01
Member No.: 664



QUOTE (Brand @ Jun 23 2011, 12:38) *
FWIW, I just ABXed a song with a 0.1 dB difference (normalized the original, which was at 0.0 to -0.1). I used headphones and focused on a very short sample repeatedly. So probably this is not the same methodology as you used, but it's still something I'd look into for such tests.

An impressive result. The only drawback is if the original was 0.0 it could possibly have caused some minor clipping. It would have been better if you had normalized to, say, -1,0 and -1.1 instead, just to be really sure that there's no clipping.

This post has been edited by Northpack: Jun 23 2011, 14:35
Go to the top of the page
+Quote Post
Kees de Visser
post Jun 23 2011, 17:44
Post #38





Group: Members
Posts: 737
Joined: 22-May 05
From: France
Member No.: 22220



QUOTE (Northpack @ Jun 23 2011, 15:34) *
An impressive result.
You can try for yourself with this online hearing test:
http://www.audiocheck.net/blindtests_index.php
Since it's relatively easy to match levels in the OP setup, 0.1 dB seems reasonable, just to eliminate yet another variable.
OTOH moving one's head will result in larger level (and spectral) differences than 0.1dB, so perhaps it's a moot point.
Go to the top of the page
+Quote Post
Northpack
post Jun 23 2011, 18:54
Post #39





Group: Members
Posts: 457
Joined: 16-December 01
Member No.: 664



QUOTE (Kees de Visser @ Jun 23 2011, 16:44) *
QUOTE (Northpack @ Jun 23 2011, 15:34) *
An impressive result.
You can try for yourself with this online hearing test:
http://www.audiocheck.net/blindtests_index.php

It seems impossible for me to ABX 0.1dB, 0.2dB only if I try very hard.
Go to the top of the page
+Quote Post
dhromed
post Jun 23 2011, 19:56
Post #40





Group: Members
Posts: 1339
Joined: 16-February 08
From: NL
Member No.: 51347



QUOTE (Kees de Visser @ Jun 23 2011, 18:44) *
QUOTE (Northpack @ Jun 23 2011, 15:34) *
An impressive result.
You can try for yourself with this online hearing test:
http://www.audiocheck.net/blindtests_index.php


Using headphones, and coming home from work after a ride in the bus and being very active in the kitchen, I dropped to 6/10 at 1dB. A second pass with more focus got me back to 10/10.
0.5dB was a guessing game. For completeness I tried 0.2 and 0.1, but I heard nothing. Pitch test got me to 10c, and then it was all the same.

The 440Hz tone made my ears bleed after a while, though.
Go to the top of the page
+Quote Post
Brand
post Jun 24 2011, 12:56
Post #41





Group: Members
Posts: 321
Joined: 27-November 09
Member No.: 75355



QUOTE (Northpack @ Jun 23 2011, 15:34) *
QUOTE (Brand @ Jun 23 2011, 12:38) *
FWIW, I just ABXed a song with a 0.1 dB difference (normalized the original, which was at 0.0 to -0.1). I used headphones and focused on a very short sample repeatedly. So probably this is not the same methodology as you used, but it's still something I'd look into for such tests.

An impressive result. The only drawback is if the original was 0.0 it could possibly have caused some minor clipping. It would have been better if you had normalized to, say, -1,0 and -1.1 instead, just to be really sure that there's no clipping.

I was thinking the same, actually. And also since one file was processed and the other wasn't. So I did as you suggested, one at -1.0 and the other at -1.1 (separate processing, of course). I managed a 8/8, then perhaps got a bit impatient, stopped at 9/11. It's not the 10/10 I'm usually aiming at, but take it as you will. I might give it another shot on a calmer day. (It's stormy right now.)
Go to the top of the page
+Quote Post
pawelq
post Jun 24 2011, 13:03
Post #42





Group: Members
Posts: 541
Joined: 20-December 05
From: Springfield, VA
Member No.: 26522



Just a question. Can you guys replicate this kind of accuracy at 0.1-0.2 dB difference if you take off your headphones and go to another room for a few minutes between trials? This is how our test was conducted (and we used loudspeakers and music, not headphones and 440 Hz tone).


--------------------
Ceterum censeo, there should be an "%is_stop_after_current%".
Go to the top of the page
+Quote Post
Brand
post Jun 24 2011, 13:54
Post #43





Group: Members
Posts: 321
Joined: 27-November 09
Member No.: 75355



I haven't tried yet, so I won't make any claims. But if I were to guess I'd say no. Even with careful listening on the headphones it wasn't easy and as I've hinted before I had to constantly repeat the involved samples to spot the differences.

But since we're scrutinizing this, what's the reasoning for only using one song for the test and this song in particular ("Falling Alice")?
Go to the top of the page
+Quote Post
pawelq
post Jun 25 2011, 05:48
Post #44





Group: Members
Posts: 541
Joined: 20-December 05
From: Springfield, VA
Member No.: 26522



QUOTE (Brand @ Jun 24 2011, 08:54) *
But since we're scrutinizing this, what's the reasoning for only using one song for the test and this song in particular ("Falling Alice")?


Only one song - mostly time constraints. We had 13 trials, add time needed for device switching, and you will realize that using multiple songs was not really feasible.

Why this song? Well, we had a pretty long discussion with very diverse suggestions, and this song was suggested by one participant mainly because of variety of timbres involved, including human voice, and relatively transparent texture. And everybody agreed to use it :-). An additional factor was that a fully analog (i.e., old enough) LP in good condition could be provided.

Most people agreed that the genre/style of music was not too unfamiliar for too many of them. Anecdotally, familiarity turned out to be a factor, as the top-scorer was the owner of the LP....


--------------------
Ceterum censeo, there should be an "%is_stop_after_current%".
Go to the top of the page
+Quote Post
Brand
post Jun 27 2011, 22:33
Post #45





Group: Members
Posts: 321
Joined: 27-November 09
Member No.: 75355



I previously kinda implied that using headphones made the ABX easier.
I've now finally tried the 0.1 dB ABX with speakers and I thought it was actually easier than with headphones. Uploaded files here.

I think usually for lossless-lossy ABXing headphones are better, but I guess with volume differences speakers could have an advantage. (Feeling the sound with the whole body etc.)


I still didn't try ABXing the same way you did for the test, though. I was repeating the short pieces back and forth. But theoretically, if the difference was in fact 0.2 dB instead of 0.1 dB, I would imagine it could be possible even with just a single listen and a bigger pause in between.
Go to the top of the page
+Quote Post
Axon
post Jun 28 2011, 21:33
Post #46





Group: Members (Donating)
Posts: 1985
Joined: 4-January 04
From: Austin, TX
Member No.: 10933



:F

If it's easier with speakers, then I'm inclined to at least suspect that the difference might be distortion-induced.
Go to the top of the page
+Quote Post
DonP
post Jun 28 2011, 23:44
Post #47





Group: Members (Donating)
Posts: 1477
Joined: 11-February 03
From: Vermont
Member No.: 4955



QUOTE (Axon @ Jun 28 2011, 15:33) *
:F

If it's easier with speakers, then I'm inclined to at least suspect that the difference might be distortion-induced.


That's the second post you've made in this thread suggesting that a small difference in volume (the other time, .05 dB) was detectable due to distortion.

By what mechanism would distortion differ significantly over a .05dB power change? Or even 1 dB?

I can certainly accept that speakers can have audible distortion. The question is how much the distortion can change, beyond just scaling with the main signal, over a very small range of level.
Go to the top of the page
+Quote Post
Dynamic
post Jun 30 2011, 00:14
Post #48





Group: Members
Posts: 833
Joined: 17-September 06
Member No.: 35307



I have a couple of queries regarding methodology (and I've only read your brief description in English so I don't know the details)

First, if there's a difference in delay, it's possible that speaker-to-stylus feedback (e.g. vibrations) could influence the sound quality in different ways. I guess high speed reel-to-reel magnetic studio tape would be an analogue alternative source that is immune to this and could be recorded (once) from the original vinyl source or real instruments.

During listening, the test organizer remained in the back of the room, invisible to the listeners.

Invisibility is certainly important, as is truly random order, so that's good procedure. It might be important that there is no other possible communication. For example, use a lamp rather than speaking to indicate the system is ready for testing and maintain a constant time-delay between auditions regardless of whether the digital component is switched into or out of use or left as the previous test (and constant time the light is lit).

A further concern with communal listening is that it's possible for some of the listeners to be influenced by each other (even subconsciously), causing their answers to be correlated rather than completely independent which can mess up the statistics and cause significance to be incorrectly inferred on a flawed assumption of statistical independence.

It's amazing how subtle things can be picked up subconsciously without trying. For example I recently heard a story of a man inadvertently made his wife think she was psychic. He was doing a really awful card trick with a friend when his wife came in (all three were staunch rationalists). The husband said "OK, guess the colour of the next card" and the friend would say "Black". Then "Right, what colour is this one?" and friend replied "Red.". They got it wrong now and then to make it seem plausible and the wife asked to have a go. Bizarrely, and unexpectedly to all concerned, she started to guess perfectly and was freaked out and thought she could be psychic or getting extra-sensory perception. She was livid with her husband when he explained what had happened. It turns out she'd subconsciously picked up the secret code without consciously realising it, and in this case it was right for red, okay for black. One might presume that the brain is reinforcing neural pathways that link various word to the same outcome and while it feels like you're picking red or black out of thin air, the tenuous neural connection can become strong enough to tip the balance of near-randomness in favour of the right answer with remarkable consistency.

It's possible that the person who switched the signal path might tap their foot subconsciously behind the curtain in a different way despite trying desperately not to influence the listeners at all.

This sounds like a good experiment, and it might be a significant result, but it can be so difficult to eliminate all possible sources of statistical correlation and effects beyond those intended for measurement. If you do test again, best of luck. It might even be interesting to rewire the Tascam internally to disconnect the line out and wire directly from the line in (with a plain analogue cable) while leaving the line in attached to the ADC if the clipping or level lights can be observed to provide a null control trial unbeknown to all participants.


--------------------
Dynamic – the artist formerly known as DickD
Go to the top of the page
+Quote Post
Arnold B. Kruege...
post Jun 30 2011, 12:18
Post #49





Group: Members
Posts: 4488
Joined: 29-October 08
From: USA, 48236
Member No.: 61311



QUOTE (pawelq @ Jun 18 2011, 18:14) *
There were 10 listeners (neither the test organizer nor me participated; he was switching the connections, I was several thousand kilometers away). All listeners listened together, being in the same room. They left the room for connection switching. During listening, the test organizer remained in the back of the room, invisible to the listeners.


The presence of a person who knew the actual status of each trial in the room is to me the most obvious weak spot in the test.

It is unecessary for him to be there, and it should have never happened.

I can hypothesize a long list of tells that may have unconsciously communicated what he knew to the people in the room. Of course I wan't there and don't know what actually happened.
Go to the top of the page
+Quote Post
RonaldDumsfeld
post Jul 1 2011, 23:29
Post #50





Group: Members
Posts: 351
Joined: 12-June 09
Member No.: 70617



Is there a possibility that some convertor chips are designed from scratch to convert at multiples of 44.1 but have the ability to resample to multiples of 48 and vice versa? Are these things supposed to be equally good at conversion (as far as the human ear is capable) at all sample rates?
Go to the top of the page
+Quote Post

3 Pages V  < 1 2 3 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 27th December 2014 - 17:32