IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
Power cable long term blind test, At hifiwigwam.com
Pio2001
post Jun 7 2006, 20:43
Post #1


Moderator


Group: Super Moderator
Posts: 3936
Joined: 29-September 01
Member No.: 73



Kiang's listening test is now finished. It was a listening test between power cords for high end hifi.

The insteresting part in this test was that every listener was at home, with his own hifi, far from the stressful conditions usually accused of making blind tests fail, with seven days to give one opinion about every cable, and to identify the hidden kettle lead. Listeners were also asked to rank the cables, and how much, in their opinion, is worth the best one.
There were 23 listeners, and the test lasted for 7 monthes.

The discussion is already 19 pages long.

Here is the start of the test : http://www.hifiwigwam.com/forum1/1614.html
Here is when the general results were published : http://www.hifiwigwam.com/forum1/1614-8.html#p113020
And here is the publication of the complete results : http://www.hifiwigwam.com/forum1/1614-19.html#p115713

The complete results include listeners comments about the cables. They are very interesting.
Here is a direct link to the results : http://www.auricles.com/Kiang_Power_cable_test(2).xls
Go to the top of the page
+Quote Post
keytotime
post Jun 7 2006, 21:53
Post #2





Group: Members
Posts: 120
Joined: 22-December 05
Member No.: 26582



LOL Score 1 for ABX
Go to the top of the page
+Quote Post
kez
post Jun 7 2006, 23:29
Post #3





Group: Members
Posts: 28
Joined: 7-April 05
Member No.: 21270



Yet another test... fine. But is it still worth trying to convince audiofools with reasonable arguments? They will still hear cables etc. no matter how many times you prove the same.
Go to the top of the page
+Quote Post
Pio2001
post Jun 7 2006, 23:48
Post #4


Moderator


Group: Super Moderator
Posts: 3936
Joined: 29-September 01
Member No.: 73



Hearing cables is an illusion. You don't convince someone that what he hears is an illusion with arguments. The only way is to show it. And to show it, a blind test is required. So the first thing to do is convincing this person to perform a blind test.

This test shows big differences perceived between twice the same sources. This phenomenon is still very seldom documented. Before this test, I had only one other link : http://www.diyaudio.com/forums/showthread....&threadid=12752
...and this listener's account is not first hand. This is just someone who tells that someone else thought that the sound was bad.

Here, for the first time, we have very accurate descriptions of strong acoustic illusions !
Go to the top of the page
+Quote Post
kez
post Jun 8 2006, 02:04
Post #5





Group: Members
Posts: 28
Joined: 7-April 05
Member No.: 21270



QUOTE (Pio2001 @ Jun 8 2006, 00:48) *
Hearing cables is an illusion. You don't convince someone that what he hears is an illusion with arguments.

OK smile.gif ... what I meant was that they will still claim that their illusion of 'hearing cables' etc. is a result of objective characteristics of that cable etc. and/or their above average hearing
Go to the top of the page
+Quote Post
Axon
post Jun 8 2006, 16:53
Post #6





Group: Members (Donating)
Posts: 1984
Joined: 4-January 04
From: Austin, TX
Member No.: 10933



Pio, you are truly a better man than I. This is a really good result, though obviously not as interesting as a positive one wink.gif

I'm not seeing too many responses in those threads. Are people accepting the results, rejecting them, or ignoring them?
Go to the top of the page
+Quote Post
krabapple
post Jun 8 2006, 19:14
Post #7





Group: Members
Posts: 2274
Joined: 18-December 03
Member No.: 10538



QUOTE (Axon @ Jun 8 2006, 11:53) *
Pio, you are truly a better man than I. This is a really good result, though obviously not as interesting as a positive one wink.gif

I'm not seeing too many responses in those threads. Are people accepting the results, rejecting them, or ignoring them?



I had a good chuckle at the results, which contained superb examples of psychological effects. But as for convincing the audiophile mass, ff123 rightly points out that the experiment lacked the statistical power (not enough subjects tested) to draw a general conclusion of 'no likely audible difference' between any of the cables (*assuming* the real audible differences are small). All one can say is that the results failed to support a hypothesis of audible difference....as did other power cable results I've seen (e.g., the trials reported by Jason Serinus on the Secrets site)

As for the wigwammer reactions, see this thread for some predictable examples of desperate data-mining, system-disparaging, requests for burn-in, etc., as the data were slowly unveiled:

http://www.hifiwigwam.com/forum1/1614-13.html

This post has been edited by krabapple: Jun 8 2006, 20:06
Go to the top of the page
+Quote Post
Pio2001
post Jun 8 2006, 21:05
Post #8


Moderator


Group: Super Moderator
Posts: 3936
Joined: 29-September 01
Member No.: 73



Reactions are quite good over there. Results were disclosed very progressively. First, the identity of the cables.
At this point, the score was not published yet, but people started to post their own individual results. Kiang was trying to sell the test account to a hifi magazine.
It is interesting to note that people who posted their results were mostly people who got the D cable right. Every people who found big differences between D and C (which were the same) were probably believing that they were the only one to have made the big mistake. So readers and participants had still a good hope that the test would succeed.
Then the scores were shown. It was a statistical failure. Now, some people were thinking that since the effect of power cables depends on the system, and since some people can hear the difference and some cannot, the test failed to confirm an effect that nonetheless exists. Especially those who heard big differences and got D=C.
Kiang was a bit reluctant to publish people's comment, fearing that it would cause trolling. But I included a warning in the Excel sheet, and advised him to PM the moderators of the board so that they monitor the discussion.
The complete results, showing strong psychological bias, came late, when every participant had already commented the general results.

The main weakness of this test is the discrepancy between the listening conditions. Every listener was listening in a different room, on a different system. If power cable could had an audible effect, in order to show it, it would be necessary to select only listeners who can hear the difference easily, and to do the test with systems on which the cable is supposed to have a strong effect.

But this is also its strenght. Previous tests were severely criticized because listeners were listening in an "unknown room", on an "unknown system", for a "very short time", in "stressful conditions". And several audiophiles says that a cable only expresses its own personality after several days of use in a perfectly known environement.

In this test, we saw the exact opposite happening : instead of allowing people to concentrate on the cable sound, the relaxed conditions allowed the psychological bias to get stronger than in usual ABX tests ! In the interconnect test done in France ( http://www.homecinema-fr.com/forum/viewtopic.php?t=29781210 ) people were barely hearing any difference, and in JV Serinus power cable test, the difference heard were moderate.
Go to the top of the page
+Quote Post
Kees de Visser
post Jun 9 2006, 09:07
Post #9





Group: Members
Posts: 677
Joined: 22-May 05
From: France
Member No.: 22220



QUOTE (Pio2001 @ Jun 8 2006, 00:48) *
This test shows big differences perceived between twice the same sources. This phenomenon is still very seldom documented.

Thanks Pio2001 for this very interesting thread.
A colleague (an audio professional) of mine objects to ABX testing because he claims that humans can't reliably identify two sources as being identical. He has no objections however to double blind testing. The result of the power cable test seem to confirm his objections.
Could it be that, when the differences between A and B become very small or even zero, psychological effects become an important (non-negligible) factor in the test results ?
I can't imagine there are no scientific studies about this effect, perhaps not in the audio field, but then ABX testing isn't exclusive to audio smile.gif Anyone who can give some sources ?
Go to the top of the page
+Quote Post
stephanV
post Jun 9 2006, 09:40
Post #10





Group: Members
Posts: 394
Joined: 6-May 04
Member No.: 13932



QUOTE (Kees de Visser @ Jun 9 2006, 10:07) *
Thanks Pio2001 for this very interesting thread.
A colleague (an audio professional) of mine objects to ABX testing because he claims that humans can't reliably identify two sources as being identical.

I'm not sure I understand his objection. Isn't ABX testing about confirming if there are differences between 2 sources, not determining if they are identical.


--------------------
"We cannot win against obsession. They care, we don't. They win."
Go to the top of the page
+Quote Post
Kees de Visser
post Jun 9 2006, 11:38
Post #11





Group: Members
Posts: 677
Joined: 22-May 05
From: France
Member No.: 22220



QUOTE (stephanV @ Jun 9 2006, 10:40) *
Isn't ABX testing about confirming if there are differences between 2 sources, not determining if they are identical.

Sure, but afaik this is done by asking the subject to decide if X is identical to A or B.
Let me emphasize that the problem only occurs when the audible differences are very small or non-existent.
I'm assuming however that ABX is mostly used to identify small differences.
Go to the top of the page
+Quote Post
stephanV
post Jun 9 2006, 12:53
Post #12





Group: Members
Posts: 394
Joined: 6-May 04
Member No.: 13932



QUOTE (Kees de Visser @ Jun 9 2006, 12:38) *
Sure, but afaik this is done by asking the subject to decide if X is identical to A or B.

I think it goes both ways. You can try to determine if X is identical to A or B, or you can try to determine if X is different from A or B. (If X is different from B, X must be equal to A).

Of course the premise that people can't determine if two sources are identical with small differences, might just be false (is there any study that confirms this?). Certainly if not is specified what a small difference is. If people can't discern small differences and such small differences are the ones that are tested, than the only logical conclusion is that most ABX tests should results in a failure. Of course this is not the case, a. because abx-ing does not only involve such small differences, and b. one could argue that not being able to reliably differentiate between two samples with very small differences is exactly what perceptual transparency is. smile.gif


--------------------
"We cannot win against obsession. They care, we don't. They win."
Go to the top of the page
+Quote Post
Pio2001
post Jun 9 2006, 14:09
Post #13


Moderator


Group: Super Moderator
Posts: 3936
Joined: 29-September 01
Member No.: 73



QUOTE (Kees de Visser @ Jun 9 2006, 10:07) *
A colleague (an audio professional) of mine objects to ABX testing because he claims that humans can't reliably identify two sources as being identical.


People's ability to identify two sources as being identical varies greatly from one listener to the next. The imagined differences range from non-existant to huge.

QUOTE (Kees de Visser @ Jun 9 2006, 10:07) *
He has no objections however to double blind testing.

The result of the power cable test seem to confirm his objections.


It depends on what he really says. How would he like to test ? With a different method than ABX, like ABA, or AB, or something like this, or does he reject the idea of repetitive sessions and just wants to perform one blinded comparison and give once for all his impressions ?
In this test, every listener just listened once and gave his results. This allowed to eliminate fatigue caused by the repetition of short stimulus. However, this weakened tremendously the statistical power of the test.
If 20 listeners perform one comparison and give their impressions, don't expect to get any significant result.

QUOTE (Kees de Visser @ Jun 9 2006, 10:07) *
Could it be that, when the differences between A and B become very small or even zero, psychological effects become an important (non-negligible) factor in the test results ?


Not really. In a blind listening test, the results are not influenced by psychological effects, since they are ruled out by the test being blind.

QUOTE (Kees de Visser @ Jun 9 2006, 10:07) *
I can't imagine there are no scientific studies about this effect, perhaps not in the audio field, but then ABX testing isn't exclusive to audio smile.gif Anyone who can give some sources ?


I was only talking about hifi, in psychology, it seems that this is well documented. Here is a good example with tomatoes smile.gif : http://www.nlh.no/ios/Publikasjoner/d2002/d2002-03.pdf

In a test such as the one you are planning ( http://www.hydrogenaudio.org/forums/index....00&#entry383100 ) I think that if you want to acheive a success, you need to study also the physical part : to measure the signal delivered by the DACs, to find what artificial signal would lead to the maximum audible effect, then to listen to such a signal, then try to find a musical sample that behaves the same.
Then the listeners can train themselves listening to both the artificial signal and the musical sample, until they can pass a blind test with the artificial signal at least, and if they can, the musical one also.
Go to the top of the page
+Quote Post
Kees de Visser
post Jun 9 2006, 15:40
Post #14





Group: Members
Posts: 677
Joined: 22-May 05
From: France
Member No.: 22220



QUOTE (stephanV @ Jun 9 2006, 13:53) *
I think it goes both ways. You can try to determine if X is identical to A or B, or you can try to determine if X is different from A or B. (If X is different from B, X must be equal to A).

Sorry for being picky: you say: "If X is different from B"... I'd say: If X is perceived different from B.
The power cable test seems to indicate that X can sound different from both A and B !
I'm trying to understand the implications of this.
QUOTE
...Certainly if not is specified what a small difference is.

You're right, sorry for being imprecise. I'd say JND (just noticable difference) or smaller.
QUOTE
If people can't discern small differences and such small differences are the ones that are tested, than the only logical conclusion is that most ABX tests should results in a failure.

Exactly, but only in the case of differences at jnd level or below.
QUOTE
Of course this is not the case, a. because abx-ing does not only involve such small differences, and b. one could argue that not being able to reliably differentiate between two samples with very small differences is exactly what perceptual transparency is. smile.gif
I don't contest the merits of ABX testing. The problem seems to be that of the 4 following possibilities, no.3 is not as unlikely as I (and many others?) have believed:
1) A and B are identical* and there is no audible difference (e.g. lossless codec)
2) A and B are different and there is no audible difference (e.g. transparent mp3)
3) A and B are identical* and there is an audible difference (psychological effects?)
4) A and B are different and there is an audible difference
(* as far as analogue stimuli can be identical)
An ABX listening test won't be able to differentiate between 1 and 2 because the differences are between JND and zero. If you're interested in that region you'll have to use other techniques like measurements or tricks to make the differences (if any) audible.
Go to the top of the page
+Quote Post
stephanV
post Jun 9 2006, 16:35
Post #15





Group: Members
Posts: 394
Joined: 6-May 04
Member No.: 13932



QUOTE (Kees de Visser @ Jun 9 2006, 16:40) *
Sorry for being picky: you say: "If X is different from B"... I'd say: If X is perceived different from B.

Being picky sometimes serves a purpose, no offense taken. smile.gif

QUOTE
The power cable test seems to indicate that X can sound different from both A and B !
I'm trying to understand the implications of this.

I'm not sure, there was only one set of ABC cables, so there was one try only. I think the probability that C was actually distinguised from D is very low looking at the test results.

Also what kind of psychological effect should be involved here? Listeners only know what cable D is and could only distinguish the other cables from the lettering AFAIK. So unless the letter C brings up certain associations with something that changes the mood, I don't know what could do that.
QUOTE
I don't contest the merits of ABX testing. The problem seems to be that of the 4 following possibilities, no.3 is not as unlikely as I (and many others?) have believed:
1) A and B are identical* and there is no audible difference (e.g. lossless codec)
2) A and B are different and there is no audible difference (e.g. transparent mp3)
3) A and B are identical* and there is an audible difference (psychological effects?)
4) A and B are different and there is an audible difference
(* as far as analogue stimuli can be identical)

I'm not sure if the nature of ABXing ever allows #3 to happen. Psychological effects also need a cause (for example knowing that cable A costs [fill-in-a-very-large-number-here]$), I don't see what in ABX testing could be that cause.


--------------------
"We cannot win against obsession. They care, we don't. They win."
Go to the top of the page
+Quote Post
Kees de Visser
post Jun 9 2006, 17:50
Post #16





Group: Members
Posts: 677
Joined: 22-May 05
From: France
Member No.: 22220



QUOTE (stephanV @ Jun 9 2006, 17:35) *
I'm not sure if the nature of ABXing ever allows #3 to happen. Psychological effects also need a cause (for example knowing that cable A costs [fill-in-a-very-large-number-here]$), I don't see what in ABX testing could be that cause.

I think you're right, the effects should average out when the test is double-blind and repetitive.
Sounds like I'm getting a bit paranoid about ABX testing. Hopefully that won't have any psychological effect on the results smile.gif

QUOTE (Pio2001 @ Jun 9 2006, 15:09)
It depends on what he really says. How would he like to test ? With a different method than ABX, like ABA, or AB, or something like this, or does he reject the idea of repetitive sessions and just wants to perform one blinded comparison and give once for all his impressions ?

We agreed about double blind and repetitive. Details are still to be discussed.
Go to the top of the page
+Quote Post
Tahnru
post Jun 9 2006, 17:57
Post #17





Group: Members
Posts: 325
Joined: 17-October 05
From: United States
Member No.: 25178



QUOTE
1) A and B are identical* and there is no audible difference (e.g. lossless codec)
2) A and B are different and there is no audible difference (e.g. transparent mp3)
3) A and B are identical* and there is an audible difference (psychological effects?)
4) A and B are different and there is an audible difference



Recall that the point of an ABX test is to gather evidence to support the hypothesis of the test: A difference exists that can be identified.

1) The hypothesis will fail - no difference can be identified (good thing for the lossless codec)
2) The hypothesis will fail - no difference can be identified (good thing for an attempt at a transparent MP3)
3) Should be accounted for by proper test design and sampling size. This is the case where ABX is explained as a way to remove guided bias in both the tester and the testee. As far as false positive results are concerned, that is why sufficient numbers of trials (and sufficient participants for a test that is more widespread) are required.
4) The hypothesis passes, and there is supposed to exist reasonable evidence to support the idea that a difference can be noticed.

The failure of an ABX test reads: "This test failed to provide evidence that an audible difference existed."

http://en.wikipedia.org/wiki/Double_blind
http://en.wikipedia.org/wiki/Codec_listening_test
http://en.wikipedia.org/wiki/Null_hypothesis
Go to the top of the page
+Quote Post
krabapple
post Jun 10 2006, 05:53
Post #18





Group: Members
Posts: 2274
Joined: 18-December 03
Member No.: 10538



QUOTE (Kees de Visser @ Jun 9 2006, 04:07) *
QUOTE (Pio2001 @ Jun 8 2006, 00:48) *
This test shows big differences perceived between twice the same sources. This phenomenon is still very seldom documented.

Thanks Pio2001 for this very interesting thread.
A colleague (an audio professional) of mine objects to ABX testing because he claims that humans can't reliably identify two sources as being identical. He has no objections however to double blind testing. The result of the power cable test seem to confirm his objections.


ABX can be considered a form of DBT, so I don't understand your colleague's objection.

QUOTE
Could it be that, when the differences between A and B become very small or even zero, psychological effects become an important (non-negligible) factor in the test results ?


Yes, that's exactly what the science of psychology tells us. It is the very REASON why tests involving small difference should be done blind.

QUOTE
I can't imagine there are no scientific studies about this effect, perhaps not in the audio field, but then ABX testing isn't exclusive to audio smile.gif Anyone who can give some sources ?



Sources that discuss psychological bias? Any good book on experimental design, sensory evaluation, or even product testing, should do.
Go to the top of the page
+Quote Post
Kees de Visser
post Jun 10 2006, 11:56
Post #19





Group: Members
Posts: 677
Joined: 22-May 05
From: France
Member No.: 22220



QUOTE (krabapple @ Jun 10 2006, 06:53) *
ABX can be considered a form of DBT, so I don't understand your colleague's objection.


I shouldn't speak for him, that would be too "lossy". He is in contact with the department of psychology of Utrecht University (Netherlands) to investigate this subject (ABX and audio) further. If there will be any results available for publication, I'll post some info here.
Go to the top of the page
+Quote Post
Pio2001
post Jun 16 2006, 23:26
Post #20


Moderator


Group: Super Moderator
Posts: 3936
Joined: 29-September 01
Member No.: 73



Alien cable discussion splitted here : http://www.hydrogenaudio.org/forums/index....ST&f=17&t=45739
Go to the top of the page
+Quote Post
Pio2001
post Aug 17 2006, 12:39
Post #21


Moderator


Group: Super Moderator
Posts: 3936
Joined: 29-September 01
Member No.: 73



The discussion was suddenly bumped in Hifiwigwam. There are some interesting posts about audio objectivism. It starts from page 21 : http://www.hifiwigwam.com/forum1/1614-21.html
Go to the top of the page
+Quote Post
kjoonlee
post Aug 17 2006, 13:05
Post #22





Group: Members
Posts: 2526
Joined: 25-July 02
From: South Korea
Member No.: 2782



OMG. I <3 Peeling.


--------------------
http://blacksun.ivyro.net/vorbis/vorbisfaq.htm
Go to the top of the page
+Quote Post
evereux
post Aug 17 2006, 13:29
Post #23





Group: Members
Posts: 907
Joined: 9-February 02
From: Cheshire, UK
Member No.: 1296



A very good read indeed. Peeling puts the points across very well. smile.gif

It's about time a programme like BBC2s Horizon covered this subject indepth. They'll have better resources in order to carry out more thorough tests. This will ofcourse help stop the general public potentially wasting monies.


--------------------
daefeatures.co.uk
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 19th September 2014 - 01:45