IPB

Welcome Guest ( Log In | Register )

3 Pages V  < 1 2 3 >  
Reply to this topicStart new topic
Controlled testing
Arnold B. Kruege...
post Feb 24 2013, 18:36
Post #26





Group: Members
Posts: 3642
Joined: 29-October 08
From: USA, 48236
Member No.: 61311



QUOTE (Yahzi @ Feb 24 2013, 11:31) *
Correct me if I'm wrong but I read an article by Sean Olive where (if I'm understanding what he says) he says that before 1994 there were no published scientific studies supporting DBT.


http://www.aes.org/e-lib/browse.cfm?elib=3839

High-Resolution Subjective Testing Using a Double-Blind Comparator

(JAES paper - peer reviewed, etc.)

"A system for the practical implementation of double-blind audibility tests is described. The controller is a self-contained unit, designed to provide setup and operational convenience while giving the user maximum sensitivity to detect differences. Standards for response matching and other controls are suggested as well as statistical methods of evaluating data. Test results to date are summarized."

The nearly identical AES conference paper was given the previous year in 1981.

I am informed that if one follows the Journal of the Acoustical Society of America, one would find papers describing earlier controlled testing relating to hearing. BTW the hearing folks use something they call an ABX test which is a little different from the one described above.
Go to the top of the page
+Quote Post
Arnold B. Kruege...
post Feb 24 2013, 18:49
Post #27





Group: Members
Posts: 3642
Joined: 29-October 08
From: USA, 48236
Member No.: 61311



QUOTE (Yahzi @ Feb 24 2013, 10:01) *
QUOTE ("Arnold B Krueger")
That's how Science works - nothing about it is totally consistent. ;-)


So you accept that sighted listening (excluding speakers) may or may not result in a null difference under controlled testing?


Of course. Sighted listening can have random results due to one or more of the many uncontrolled influences. Doesn't happen very often because the strongest influences usually produce consistent results.

QUOTE
QUOTE
The operative word appears to be give, which ordinarily implies altruism on the part of the provider and an exploitative situation with the person making the demands.


Perhaps you did not take what I said in the spirit that I intended. What I meant was, it would be great to have a compiled list of CDP .. or amp or cable tests to serve as ammunition in these arguments - to have convincing evidence.


Such lists exist. A little use of google turns them up.

QUOTE
I've heard on a number of forums where proponents of DBT will say something to the effect of "well, show me a single positive DBT of a CDP" or something like that. Or show me a positive result of speaker cable ..or amplifiers ... etc. If at least a few positive tests exist (excluding speakers here) for everything else then what does that mean?


I know how to do a positive ABX test involving a CD player. In fact one is described in this article: Masters, Ian G 'Do All CD Players Sound the Same?' Stereo review, Jan 1986, pg 50-57. I was part of the listening panel and I reviewed the test setup and approved of it. The player had measurable faults that were obvious enough to make an audible difference with very critical program material.
Go to the top of the page
+Quote Post
DonP
post Feb 24 2013, 19:12
Post #28





Group: Members (Donating)
Posts: 1471
Joined: 11-February 03
From: Vermont
Member No.: 4955



QUOTE (Arnold B. Krueger @ Feb 23 2013, 14:18) *
If you want the most sensitive results, you have to have some kind of fast switching available. Some authorities also want a transient-free switch which involves modulating the signal. Putting the actual switching under the control of the listener also maximizes sensitivity.


If what you are testing is someone's claim that he can tell a difference in the sound of 2 components auditioned on different days in different buildings, then you don't need fast switching.



Go to the top of the page
+Quote Post
mzil
post Feb 24 2013, 19:35
Post #29





Group: Members
Posts: 497
Joined: 5-August 07
Member No.: 45913



QUOTE (Yahzi @ Feb 24 2013, 10:01) *
[What I meant was, it would be great to have a compiled list of CDP .. or amp or cable tests to serve as ammunition in these arguments - to have convincing evidence


Have you already seen this and this?
Go to the top of the page
+Quote Post
Arnold B. Kruege...
post Feb 24 2013, 20:07
Post #30





Group: Members
Posts: 3642
Joined: 29-October 08
From: USA, 48236
Member No.: 61311



QUOTE (DonP @ Feb 24 2013, 13:12) *
QUOTE (Arnold B. Krueger @ Feb 23 2013, 14:18) *
If you want the most sensitive results, you have to have some kind of fast switching available. Some authorities also want a transient-free switch which involves modulating the signal. Putting the actual switching under the control of the listener also maximizes sensitivity.


If what you are testing is someone's claim that he can tell a difference in the sound of 2 components auditioned on different days in different buildings, then you don't need fast switching.


The above ignores a clearly stated condition: "If you want the most sensitive results". If you audition on different days in different buildings then you don't want the most sensitive results.
Go to the top of the page
+Quote Post
greynol
post Feb 24 2013, 20:22
Post #31





Group: Super Moderator
Posts: 10000
Joined: 1-April 04
From: San Francisco
Member No.: 13167



Agreed, although that doesn't help to address the placebophile argument that X is less fatiguing than Y or is more beneficial to the mental, psychological and/or physical health of the listener (yes, still confined to the realm of listening to audio).

I guess it doesn't matter since this generally boils down to metaphysical beliefs, even if the individual espousing them insists otherwise.


--------------------
YOUR EYES CANNOT HEAR!!!!!!!!!!!
Go to the top of the page
+Quote Post
Yahzi
post Feb 24 2013, 20:40
Post #32





Group: Members
Posts: 131
Joined: 21-November 12
Member No.: 104669



QUOTE ("mzil")
Have you already seen this and this?


I saw on this site many of the DBTs are in French .. some links don't work and all the tests on the AES or Stereo Review can't be accessed. I see one CDP DBT with a failed link ... and the rest I see are in French. One failed, one even was a success ...
Go to the top of the page
+Quote Post
Yahzi
post Feb 24 2013, 21:34
Post #33





Group: Members
Posts: 131
Joined: 21-November 12
Member No.: 104669



If the DBT results were published at the AES does that make them peer-reviewed?
Go to the top of the page
+Quote Post
Arnold B. Kruege...
post Feb 25 2013, 14:41
Post #34





Group: Members
Posts: 3642
Joined: 29-October 08
From: USA, 48236
Member No.: 61311



QUOTE (Yahzi @ Feb 24 2013, 15:34) *
If the DBT results were published at the AES does that make them peer-reviewed?


Everything published in the JAES as a technical paper is AFAIK still heartily peer-reviewed.

Almost anything can be said in a conference paper and they usually get published separately from the Journal.

There are a few JAES articles that some sophisticated and influential members would like to peer review err, retroactively. Many of them relate to exceptional claims of the audibility of Transient Intermodulation or Slew Induced distortion.

Doing your own DBTs isn't exceptionally difficult nor does it take a lot of rare resources if you are willing to be pragmatic.

This post has been edited by Arnold B. Krueger: Feb 25 2013, 14:46
Go to the top of the page
+Quote Post
mzil
post Feb 25 2013, 20:34
Post #35





Group: Members
Posts: 497
Joined: 5-August 07
Member No.: 45913



^To add to that I'd like to say that having done it myself (a layman without any schooling in these disciplines), after I made the wager with my my golden-eared audiophile friend, I felt vindicated in my beliefs and now when asked, "But have you ever conducted such carefully controlled tests yourself?" I can proudly respond in the affirmative. smile.gif

It is quite empowering Yahzi. Give it a try!

Go to the top of the page
+Quote Post
Yahzi
post Feb 25 2013, 21:18
Post #36





Group: Members
Posts: 131
Joined: 21-November 12
Member No.: 104669



QUOTE ("mzil")
It is quite empowering Yahzi. Give it a try!


I want to! But first I need to carefully study the necessary steps involved. emot-v.gif Without an ABX comparator I would be at a disadvantage .. I would imagine. Although I meant what I said, that I would love to meet some of you guys in the near future and perhaps learn more about the testing in person and even be tested myself.
Go to the top of the page
+Quote Post
Arnold B. Kruege...
post Feb 26 2013, 02:01
Post #37





Group: Members
Posts: 3642
Joined: 29-October 08
From: USA, 48236
Member No.: 61311



QUOTE (Yahzi @ Feb 25 2013, 15:18) *
QUOTE ("mzil")
It is quite empowering Yahzi. Give it a try!


I want to! But first I need to carefully study the necessary steps involved. emot-v.gif Without an ABX comparator I would be at a disadvantage .. I would imagine. Although I meant what I said, that I would love to meet some of you guys in the near future and perhaps learn more about the testing in person and even be tested myself.


Just about everybody can have an ABX Comparator on their computer - they exist as free downloads for just about every kind of computer that runs Java.

Many things of interest including just about everything related to lossy compression can be tested this way.

While it takes a step of faith in the quality of the converters that can be attached to computers, most issues that relate to hardware can also be recast as files for a software ABX Comparator.

Here is an archive of files for use with a software ABX Comparagtor:

http://www.ethanwiner.com/aes/
Go to the top of the page
+Quote Post
Yahzi
post Feb 28 2013, 12:52
Post #38





Group: Members
Posts: 131
Joined: 21-November 12
Member No.: 104669



Arny, I noticed on occasion you bring up time synchronisation. Can you please explain what that entails and the importance of doing it? How does one time synch and if you don't do this, would it invalidate the blind or double blind test results?

This post has been edited by Yahzi: Feb 28 2013, 12:52
Go to the top of the page
+Quote Post
pdq
post Feb 28 2013, 15:29
Post #39





Group: Members
Posts: 3371
Joined: 1-September 05
From: SE Pennsylvania
Member No.: 24233



It's not that complicated. When you switch rapidly between two audio streams, if they are not exactly synchronized then you will hear a little "hiccup" from either a few samples being repeated, or a few being skipped, also probably a slight click as you switch. This tells you if X is matched to A or B, even if A and B are identical except for the time mismatch.
Go to the top of the page
+Quote Post
Yahzi
post Feb 28 2013, 16:45
Post #40





Group: Members
Posts: 131
Joined: 21-November 12
Member No.: 104669



Okay, but if the streams are not synched ... that will invalidate the test results? I understand quick switching ... but I've seen many tests that are not time synched, so I'm just asking here whether that would be enough to throw the results in the air.
Go to the top of the page
+Quote Post
Arnold B. Kruege...
post Feb 28 2013, 20:37
Post #41





Group: Members
Posts: 3642
Joined: 29-October 08
From: USA, 48236
Member No.: 61311



QUOTE (Yahzi @ Feb 28 2013, 10:45) *
Okay, but if the streams are not synched ... that will invalidate the test results? I understand quick switching ... but I've seen many tests that are not time synched, so I'm just asking here whether that would be enough to throw the results in the air.


If you don't synchronize the streams it is possible to correctly identify them, even if they are otherwise identical.

The test isn't really blind.

This post has been edited by Arnold B. Krueger: Feb 28 2013, 20:37
Go to the top of the page
+Quote Post
Yahzi
post Mar 15 2013, 10:22
Post #42





Group: Members
Posts: 131
Joined: 21-November 12
Member No.: 104669



So double blind testing only really works if you are experienced in it, i.e. having trained yourself to do it, knowing what to listen for. So if you lure inexperienced DBT 'testers' into the test you could invalidate the results that way. Surely?
Go to the top of the page
+Quote Post
Arnold B. Kruege...
post Mar 15 2013, 20:28
Post #43





Group: Members
Posts: 3642
Joined: 29-October 08
From: USA, 48236
Member No.: 61311



QUOTE (Yahzi @ Mar 15 2013, 05:22) *
So double blind testing only really works if you are experienced in it, i.e. having trained yourself to do it, knowing what to listen for. So if you lure inexperienced DBT 'testers' into the test you could invalidate the results that way. Surely?


Obvious anti DBT bias noted. In fact listening tests of any kind only work if you have experienced listeners. In general sighted evaluations don't work because of the unresolved problems with listener bias.
Go to the top of the page
+Quote Post
Porcus
post Mar 15 2013, 23:31
Post #44





Group: Members
Posts: 1842
Joined: 30-November 06
Member No.: 38207



QUOTE (Yahzi @ Mar 15 2013, 10:22) *
So double blind testing only really works if you are experienced in it, i.e. having trained yourself to do it, knowing what to listen for. So if you lure inexperienced DBT 'testers' into the test you could invalidate the results that way. Surely?


With 'testers', do you mean the person(s) who listen, or the person(s) who administer the test and collect the assessments from the former?

If you 'lure' someone inexperienced with listening into a listening test, then they might not spot differences they would if they were trained, but that is the inexperience with the 'T' part, not the 'DB' part.

If you 'lure' someone inexperienced (... dare I say clueless?) into setting up a DBT, that person may of course invalidate the test by making errors an experienced tester would avoid. Those errors could be related to double-blind framework (for example, not everyone knows what that 'double' is about), or unrelated (e.g. not matching volume, and making the wrong interpretations).


--------------------
One day in the Year of the Fox came a time remembered well
Go to the top of the page
+Quote Post
krabapple
post Mar 16 2013, 05:38
Post #45





Group: Members
Posts: 2181
Joined: 18-December 03
Member No.: 10538



QUOTE (Yahzi @ Mar 15 2013, 05:22) *
So double blind testing only really works if you are experienced in it, i.e. having trained yourself to do it, knowing what to listen for. So if you lure inexperienced DBT 'testers' into the test you could invalidate the results that way. Surely?


If a person already claims to hear a difference between A and B, a blind test of that person is an excellent way to test them on that claim.

If you are a researcher and want to determine if humans could hear a difference between A and B, then training is part of the experimental protocol.

Surely you are being purposely obtuse?

This post has been edited by krabapple: Mar 16 2013, 05:40
Go to the top of the page
+Quote Post
Yahzi
post Mar 25 2013, 17:30
Post #46





Group: Members
Posts: 131
Joined: 21-November 12
Member No.: 104669



I was just being curious and sometimes curiosity gets the better of me. smile.gif I've been told on several occasions that human auditory memory is very short. Are there any studies I can look at that can confirm this? Thanks.
Go to the top of the page
+Quote Post
Arnold B. Kruege...
post Mar 25 2013, 20:07
Post #47





Group: Members
Posts: 3642
Joined: 29-October 08
From: USA, 48236
Member No.: 61311



QUOTE (Yahzi @ Mar 25 2013, 12:30) *
I was just being curious and sometimes curiosity gets the better of me. smile.gif I've been told on several occasions that human auditory memory is very short. Are there any studies I can look at that can confirm this? Thanks.


The topic is discussed pretty thoroughly with extensive footnotes in This Is Your Brain On Music by Daniel Levitin

http://www.amazon.com/This-Your-Brain-Musi...n/dp/0452288525

The short answer is if the goal is to remember small subtle details, probably a second or less. Obviously, gross details such as the name of the song or whether we liked the performance can be recalled for months or years.

I built an ABX Comparator with an adjustable delay while switching. In general my ability to distinguish small differences becomes degraded when this delay is much more than a second.

This post has been edited by Arnold B. Krueger: Mar 25 2013, 20:08
Go to the top of the page
+Quote Post
Yahzi
post Mar 25 2013, 20:11
Post #48





Group: Members
Posts: 131
Joined: 21-November 12
Member No.: 104669



Thanks Arnold. Are there any other books or studies you would recommend? I've already ordered "This Is Your Brain On Music" but I would like to know if there are any papers, or research you could recommend I read in addition. Thanks again!
Go to the top of the page
+Quote Post
krabapple
post Mar 26 2013, 19:03
Post #49





Group: Members
Posts: 2181
Joined: 18-December 03
Member No.: 10538



see this thread

http://www.hydrogenaudio.org/forums/index.php?showtopic=7645
Go to the top of the page
+Quote Post
Yahzi
post Mar 26 2013, 20:47
Post #50





Group: Members
Posts: 131
Joined: 21-November 12
Member No.: 104669



Yeah I saw the thread but I see a dozen different books. How do I know that there is anything of value specifically concerning auditory memory in there? I have no idea. That's why I'm asking so I can be pointed in the right direction. I can't order all these books you know. biggrin.gif
Go to the top of the page
+Quote Post

3 Pages V  < 1 2 3 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 23rd July 2014 - 14:43