IPB

Welcome Guest ( Log In | Register )

What's the problem with double-blind testing?
mirrorsawlljk
post Oct 19 2005, 03:03
Post #1





Group: Members
Posts: 11
Joined: 15-October 05
Member No.: 25138



I happened to pick up an issue of stereophile at a record store I visited and I was pretty shocked to see a seemingly intelligent person in the correspondence section bashing double blind testing as being unreliable. I'm afraid I don't understand his angle of attack. I don't see how anything could be a more reliable test of sound quality differences than a properly conducted double blind listening test.

I'm almost afraid to read the rest of the magazine if this is the kind of letter they think is worth publishing. Is there an audio magazine that isn't filled with this kind of thinking?
Go to the top of the page
+Quote Post
 
Start new topic
Replies
onthejazz
post Oct 19 2005, 10:03
Post #2





Group: Members
Posts: 117
Joined: 2-April 02
From: Minnesota
Member No.: 1666



Interesting publication. I like it, too bad its not in full production anymore. Thanks much for the link. Anybody have any more quality sources?
Go to the top of the page
+Quote Post
Donunus
post Oct 19 2005, 11:00
Post #3





Group: Members
Posts: 226
Joined: 8-July 05
Member No.: 23210



The writer of the letter in stereophile is incorrect in saying double blind tests are unreliable but he has a point. I do abx tests in foobar to compare different files a lot but there just are too many songs to abx to make sure all of one formats files are as good as the other ex. mp3 vs aac vs wav, etc... The thing I always notice though is that when using the analytical side of the brain(abx testing) only certain parts of the music is being focused and the emotional side of the music isn't being analyzed, therefore the goosebump factor of the sound the way the music delivers emotion cannot be analyzed! Its very hard to have the left and right side of the brain working at the same time.

My Example for this is when doing an abx of one particular song, I passed the test in foobar with flying colors(100% on abx test) but thought the differences were not really significant enough to make me keep the wav files from the cds and kept only the mp3s. After a few weeks of listening to the mp3s and really getting to know and love the songs better, I tried just popping the cd in for a listen... wow, goosebumps.... The parts I loved in the songs i was listening to gave me goosebumps for the first time. Now thats music! The feeling is lost on some parts of the mp3. And that was with my pc using sennheisers, not even my high end home rig.

Its not totally reliable to trust abx testing for determining the enjoyment one gets from his music because human memory can only remember only a certain number of seconds at a time like in an abx test. Its good for making codecs for eliminating artifacts but abx testing a few seconds of one song does not determine the accuracy of the dynamics of the whole piece. All the buildup of sound, the emotion! Bash me now but I have been an audiophile for more than 15 years and I am only 31, and have been a music lover since 5 years old. I know what I'm talking about. Listening to a bad stereo system, I can hear differences between different files for example but I don't really tend to care whether I am listening to the mp3 or the wav version. But with a very involving music system, the mp3 can sometimes put a damper on the fun factor of the song.

Also, stereophile loves posting "gray area" letters. I love that magazine. They exaggerate things sometimes but overall they are very accurate the way they subjectively describe the audio quality of equipment.
Go to the top of the page
+Quote Post
Danimal
post Oct 19 2005, 20:20
Post #4





Group: Members
Posts: 132
Joined: 16-June 03
From: Ohio
Member No.: 7209



QUOTE (Donunus @ Oct 19 2005, 05:00 AM)
The writer of the letter in stereophile is incorrect in saying double blind tests are unreliable but he has a point. I do abx tests in foobar to compare different files a lot but there just are too many songs to abx to make sure all of one formats files are as good as the other ex. mp3 vs aac vs wav, etc... The thing I always notice though is that when using the analytical side of the brain(abx testing) only certain parts of the music is being focused and the emotional side of the music isn't being analyzed, therefore the goosebump factor of the sound the way the music delivers emotion cannot be analyzed! Its very hard to have the left and right side of the brain working at the same time.

My Example for this is when doing an abx of one particular song, I passed the test in foobar with flying colors(100% on abx test) but thought the differences were not really significant enough to make me keep the wav files from the cds and kept only the mp3s. After a few weeks of listening to the mp3s and really getting to know and love the songs better, I tried just popping the cd in for a listen... wow, goosebumps.... The parts I loved in the songs i was listening to gave me goosebumps for the first time. Now thats music! The feeling is lost on some parts of the mp3. And that was with my pc using sennheisers, not even my high end home rig.

Its not totally reliable to trust abx testing for determining the enjoyment one gets from his music because human memory can only remember only a certain number of seconds at a time like in an abx test. Its good for making codecs for eliminating artifacts but abx testing a few seconds of one song does not determine the accuracy of the dynamics of the whole piece. All the buildup of sound, the emotion! Bash me now but I have been an audiophile for more than 15 years and I am only 31, and have been a music lover since 5 years old. I know what I'm talking about. Listening to a bad stereo system, I can hear differences between different files for example but I don't really tend to care whether I am listening to the mp3 or the wav version. But with a very involving music system, the mp3 can sometimes put a damper on the fun factor of the song.

Also, stereophile loves posting "gray area" letters. I love that magazine. They exaggerate things sometimes but overall they are very accurate the way they subjectively describe the audio quality of equipment.
*


The point of double-blind testing is to make sure that you're getting goosebumps over something you actually hear, and not something you expect to hear. The whole point is to try to evaluate what you're hearing without knowing whether it's the original or the lossy encode. That shouldn't remove the "emotional" part of it, just make sure that you're reacting based upon something that is really there, and not what you want to hear or what you expect to hear.
Go to the top of the page
+Quote Post
Donunus
post Oct 20 2005, 01:51
Post #5





Group: Members
Posts: 226
Joined: 8-July 05
Member No.: 23210



QUOTE (Danimal @ Oct 20 2005, 03:20 AM)
QUOTE (Donunus @ Oct 19 2005, 05:00 AM)
The writer of the letter in stereophile is incorrect in saying double blind tests are unreliable but he has a point. I do abx tests in foobar to compare different files a lot but there just are too many songs to abx to make sure all of one formats files are as good as the other ex. mp3 vs aac vs wav, etc... The thing I always notice though is that when using the analytical side of the brain(abx testing) only certain parts of the music is being focused and the emotional side of the music isn't being analyzed, therefore the goosebump factor of the sound the way the music delivers emotion cannot be analyzed! Its very hard to have the left and right side of the brain working at the same time.

My Example for this is when doing an abx of one particular song, I passed the test in foobar with flying colors(100% on abx test) but thought the differences were not really significant enough to make me keep the wav files from the cds and kept only the mp3s. After a few weeks of listening to the mp3s and really getting to know and love the songs better, I tried just popping the cd in for a listen... wow, goosebumps.... The parts I loved in the songs i was listening to gave me goosebumps for the first time. Now thats music! The feeling is lost on some parts of the mp3. And that was with my pc using sennheisers, not even my high end home rig.

Its not totally reliable to trust abx testing for determining the enjoyment one gets from his music because human memory can only remember only a certain number of seconds at a time like in an abx test. Its good for making codecs for eliminating artifacts but abx testing a few seconds of one song does not determine the accuracy of the dynamics of the whole piece. All the buildup of sound, the emotion! Bash me now but I have been an audiophile for more than 15 years and I am only 31, and have been a music lover since 5 years old. I know what I'm talking about. Listening to a bad stereo system, I can hear differences between different files for example but I don't really tend to care whether I am listening to the mp3 or the wav version. But with a very involving music system, the mp3 can sometimes put a damper on the fun factor of the song.

Also, stereophile loves posting "gray area" letters. I love that magazine. They exaggerate things sometimes but overall they are very accurate the way they subjectively describe the audio quality of equipment.
*


The point of double-blind testing is to make sure that you're getting goosebumps over something you actually hear, and not something you expect to hear. The whole point is to try to evaluate what you're hearing without knowing whether it's the original or the lossy encode. That shouldn't remove the "emotional" part of it, just make sure that you're reacting based upon something that is really there, and not what you want to hear or what you expect to hear.
*



Well, with mp3, even 320. I have passed foobar abx with many samples from my own cds compared to the original with flying colors. What I am saying is I cannot get goosebumps while I am in the analyzing mode cause I am not really listening to the music but to different aspects of the sound while doing the abx test. I do hear the differences and although they are not really that big from an analysis standpoint, they become bigger when listening for enjoyment. Some songs will just lose some life when encoded to mp3.
Go to the top of the page
+Quote Post
stephanV
post Oct 20 2005, 08:48
Post #6





Group: Members
Posts: 394
Joined: 6-May 04
Member No.: 13932



QUOTE (Donunus @ Oct 20 2005, 02:51 AM)
Well, with mp3, even 320. I have passed foobar abx with many samples from my own cds compared to the original with flying colors. What I am saying is I cannot get goosebumps while I am in the analyzing mode cause I am not really listening to the music but to different aspects of the sound while doing the abx test. I do hear the differences and although they are not really that big from an analysis standpoint, they become bigger when listening for enjoyment. Some songs will just lose some life when encoded to mp3.
*

What you are saying is contradictional. You say that when you are paying close attention to the music you hear less difference than when you do casual listening. That is very odd and seems to be more related to expectation.


--------------------
"We cannot win against obsession. They care, we don't. They win."
Go to the top of the page
+Quote Post
bryant
post Oct 20 2005, 18:11
Post #7


WavPack Developer


Group: Developer (Donating)
Posts: 1290
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



QUOTE (stephanV @ Oct 19 2005, 11:48 PM)
QUOTE (Donunus @ Oct 20 2005, 02:51 AM)
Well, with mp3, even 320. I have passed foobar abx with many samples from my own cds compared to the original with flying colors. What I am saying is I cannot get goosebumps while I am in the analyzing mode cause I am not really listening to the music but to different aspects of the sound while doing the abx test. I do hear the differences and although they are not really that big from an analysis standpoint, they become bigger when listening for enjoyment. Some songs will just lose some life when encoded to mp3.
*

What you are saying is contradictional. You say that when you are paying close attention to the music you hear less difference than when you do casual listening. That is very odd and seems to be more related to expectation.
*



It turns out that the science of perception is rich with results that run counter to common sense. Think how wrong common sense is when trying to understand quantum mechanics! Well, it is equally bad when trying to understand cognitive psychology and perception. To tell you the truth, I cannot understand how someone could read and understand the two papers I linked to in the following thread and still be convinced that DBT can be the final word on music reproduction. In any event, it is a very interesting field of science that I suspect is not well known to many HA members:

http://www.hydrogenaudio.org/forums/index....ndpost&p=321782
Go to the top of the page
+Quote Post

Posts in this topic
- mirrorsawlljk   What's the problem with double-blind testing?   Oct 19 2005, 03:03
- - bubka   some people can actually detect specific codecs by...   Oct 19 2005, 03:07
- - TheQat   QUOTE (mirrorsawlljk @ Oct 18 2005, 06:03 PM)...   Oct 19 2005, 03:27
|- - singaiya   QUOTE (TheQat @ Oct 18 2005, 06:27 PM)QUOTE (...   Oct 19 2005, 04:31
|- - Yaztromo   QUOTE (TheQat @ Oct 19 2005, 03:27 AM)Edit: H...   Oct 19 2005, 23:05
- - Tahnru   The closest thing I have seen to a legitimate crit...   Oct 19 2005, 04:29
- - Axon   The Audio Critic is notably pro-DBT.   Oct 19 2005, 07:06
|- - PoisonDan   QUOTE (Axon @ Oct 19 2005, 08:06 AM)The Audio...   Oct 19 2005, 12:18
- - onthejazz   Interesting publication. I like it, too bad its no...   Oct 19 2005, 10:03
|- - Donunus   The writer of the letter in stereophile is incorre...   Oct 19 2005, 11:00
|- - Danimal   QUOTE (Donunus @ Oct 19 2005, 05:00 AM)The wr...   Oct 19 2005, 20:20
|- - Donunus   QUOTE (Danimal @ Oct 20 2005, 03:20 AM)QUOTE ...   Oct 20 2005, 01:51
|- - stephanV   QUOTE (Donunus @ Oct 20 2005, 02:51 AM)Well, ...   Oct 20 2005, 08:48
||- - bryant   QUOTE (stephanV @ Oct 19 2005, 11:48 PM)QUOTE...   Oct 20 2005, 18:11
|||- - Pio2001   QUOTE (bryant @ Oct 20 2005, 07:11 PM)To tell...   Oct 20 2005, 20:48
||||- - singaiya   Thanks for the papers, David. The first one is a g...   Oct 20 2005, 22:13
||||- - duff   QUOTE (Pio2001 @ Oct 20 2005, 07:48 PM)QUOTE ...   Oct 20 2005, 22:52
||||- - stephanV   QUOTE (duff @ Oct 20 2005, 11:52 PM)It's ...   Oct 20 2005, 23:11
|||||- - duff   QUOTE Might be... or might be not. You can't c...   Oct 20 2005, 23:36
|||||- - KikeG   QUOTE (duff @ Oct 20 2005, 11:36 PM)I think i...   Oct 21 2005, 00:00
||||||- - rjamorim   QUOTE (KikeG @ Oct 20 2005, 09:00 PM)Also, ps...   Oct 21 2005, 00:15
||||||- - duff   QUOTE (KikeG @ Oct 20 2005, 11:00 PM)QUOTE (d...   Oct 21 2005, 00:29
||||||- - rjamorim   QUOTE (duff @ Oct 20 2005, 09:29 PM)I'm n...   Oct 21 2005, 01:03
||||||- - Mike Giacomelli   QUOTE (duff @ Oct 20 2005, 04:29 PM)The audit...   Oct 21 2005, 05:56
||||||- - KikeG   QUOTE (duff @ Oct 21 2005, 12:29 AM)From the ...   Oct 21 2005, 08:10
|||||- - stephanV   QUOTE (duff @ Oct 21 2005, 12:36 AM)I think i...   Oct 21 2005, 08:34
|||||- - Woodinville   QUOTE (stephanV @ Oct 20 2005, 11:34 PM)Sorry...   Oct 21 2005, 21:19
||||- - krabapple   QUOTE (duff @ Oct 20 2005, 04:52 PM)QUOTE (Pi...   Oct 21 2005, 05:25
||||- - antz   QUOTE (duff @ Oct 20 2005, 10:52 PM)QUOTE (Pi...   Oct 21 2005, 13:39
||||- - Woodinville   QUOTE (antz @ Oct 21 2005, 04:39 AM)Seems to ...   Oct 21 2005, 21:20
|||- - stephanV   QUOTE (bryant @ Oct 20 2005, 07:11 PM)It turn...   Oct 20 2005, 21:43
||- - Donunus   QUOTE (stephanV @ Oct 20 2005, 03:48 PM)QUOTE...   Oct 21 2005, 04:10
||- - stephanV   QUOTE (Donunus @ Oct 21 2005, 05:10 AM)I do n...   Oct 21 2005, 08:43
||- - user   I recall, we have had successful abx tests even he...   Oct 21 2005, 09:20
||- - Pio2001   QUOTE (user @ Oct 21 2005, 10:20 AM)I recall,...   Oct 21 2005, 12:18
|||- - ff123   QUOTE (Pio2001 @ Oct 21 2005, 03:18 AM)QUOTE ...   Oct 21 2005, 17:43
||- - krabapple   QUOTE (user @ Oct 21 2005, 03:20 AM)I recall,...   Oct 21 2005, 21:54
|- - Cartoon   QUOTE (Donunus @ Oct 20 2005, 02:51 AM)I do h...   Nov 12 2005, 13:40
- - KikeG   The "goosebump", emotional factor can be...   Oct 19 2005, 11:54
|- - Donunus   QUOTE (KikeG @ Oct 19 2005, 06:54 PM)The ...   Oct 19 2005, 12:06
|- - kjoonlee   QUOTE (Donunus @ Oct 19 2005, 08:06 PM)actual...   Oct 19 2005, 18:31
||- - Woodinville   QUOTE (kjoonlee @ Oct 19 2005, 09:31 AM)QUOTE...   Oct 21 2005, 01:27
||- - duff   QUOTE If you consider that we are wired to detect ...   Oct 21 2005, 01:38
||- - Woodinville   QUOTE (duff @ Oct 20 2005, 04:38 PM)So we are...   Oct 21 2005, 21:15
||- - duff   QUOTE (Woodinville @ Oct 21 2005, 08:15 PM)QU...   Oct 21 2005, 23:43
||- - Woodinville   QUOTE (duff @ Oct 21 2005, 02:43 PM)When I sa...   Oct 22 2005, 08:00
|- - Lyx   QUOTE (Donunus @ Oct 19 2005, 01:06 PM)QUOTE ...   Oct 20 2005, 06:07
|- - Donunus   QUOTE (Lyx @ Oct 20 2005, 01:07 PM)QUOTE (Don...   Oct 20 2005, 16:51
|- - Lyx   QUOTE (Donunus @ Oct 20 2005, 05:51 PM)QUOTE ...   Oct 20 2005, 21:10
- - KikeG   Well, I think expectation effects in listening tes...   Oct 19 2005, 19:33
|- - krabapple   QUOTE (KikeG @ Oct 19 2005, 10:33 AM)Well, I ...   Oct 19 2005, 20:12
- - krabapple   QUOTE (mirrorsawlljk @ Oct 18 2005, 06:03 PM)...   Oct 19 2005, 20:06
- - Halcyon   There is nothing inherently bad about double blind...   Oct 19 2005, 20:29
- - ff123   Tests that try to distinguish very small effects *...   Oct 19 2005, 20:53
|- - krabapple   QUOTE (ff123 @ Oct 19 2005, 11:53 AM)Tests th...   Oct 19 2005, 23:40
|- - ff123   QUOTE (krabapple @ Oct 19 2005, 02:40 PM)Inte...   Oct 20 2005, 05:38
- - fcmts   There is another problem similar to wine blind tes...   Oct 19 2005, 22:15
- - stephanV   I don't see how that is a problem. The goal of...   Oct 19 2005, 22:22
- - duff   One distinction relevant to this issue is the diff...   Oct 20 2005, 23:00
- - duff   QUOTE The opposite of "blind", I suppose...   Oct 21 2005, 01:27
- - Jun-Dai   I think it's pretty clear that double-blind te...   Oct 21 2005, 07:22
- - KikeG   For helping doing long-term, casual listening-like...   Oct 21 2005, 08:30
- - duff   QUOTE Seems to be a lack of understanding of somet...   Oct 21 2005, 16:48
|- - stephanV   QUOTE (duff @ Oct 21 2005, 05:48 PM)The reaso...   Oct 21 2005, 17:29
||- - duff   QUOTE (stephanV @ Oct 21 2005, 04:29 PM)QUOTE...   Oct 21 2005, 20:06
||- - stephanV   QUOTE (duff @ Oct 21 2005, 09:06 PM)QUOTE Wh...   Oct 21 2005, 20:21
||- - bryant   QUOTE (stephanV @ Oct 21 2005, 11:21 AM)QUOTE...   Oct 21 2005, 22:53
||- - rjamorim   QUOTE (bryant @ Oct 21 2005, 07:53 PM)But I d...   Oct 21 2005, 23:12
||- - stephanV   QUOTE (bryant @ Oct 21 2005, 11:53 PM)So, if ...   Oct 22 2005, 00:00
||- - ChiGung   QUOTE (bryant @ Oct 21 2005, 10:53 PM)What...   Oct 22 2005, 00:25
|||- - bryant   QUOTE (ChiGung @ Oct 21 2005, 03:25 PM)I have...   Oct 22 2005, 23:39
|||- - ChiGung   just thinking -i do get all sorts of weird sensati...   Oct 23 2005, 00:34
||- - Woodinville   QUOTE (bryant @ Oct 21 2005, 01:53 PM)So, if ...   Oct 22 2005, 07:52
||- - bryant   QUOTE (Woodinville @ Oct 21 2005, 10:52 PM)QU...   Oct 22 2005, 23:32
||- - Woodinville   QUOTE (bryant @ Oct 22 2005, 02:32 PM)The onl...   Oct 24 2005, 21:51
|- - Lyx   QUOTE (duff @ Oct 21 2005, 05:48 PM)QUOTE See...   Oct 21 2005, 17:35
|- - krabapple   QUOTE (duff @ Oct 21 2005, 10:48 AM)QUOTE See...   Oct 21 2005, 22:01
- - duff   QUOTE It sounds counterintuitive that my brain wou...   Oct 22 2005, 00:30
|- - ChiGung   QUOTE (duff @ Oct 22 2005, 12:30 AM)Frankly, ...   Oct 22 2005, 00:47
||- - duff   QUOTE (ChiGung @ Oct 21 2005, 11:47 PM)QUOTE ...   Oct 22 2005, 01:14
||- - ChiGung   QUOTE (duff @ Oct 22 2005, 01:14 AM)The reduc...   Oct 22 2005, 01:26
|- - Woodinville   QUOTE (duff @ Oct 21 2005, 03:30 PM)Filtering...   Oct 22 2005, 08:07
- - ChiGung   A simplistic example - a sine wave of exactly 3122...   Oct 22 2005, 00:59
|- - Woodinville   QUOTE (ChiGung @ Oct 21 2005, 03:59 PM)A simp...   Oct 22 2005, 08:08
|- - ChiGung   QUOTE (Woodinville @ Oct 22 2005, 08:08 AM)QU...   Oct 22 2005, 14:09
- - ChiGung   -Sorry for flooding a bit there. Tasty subject   Oct 22 2005, 01:01
- - duff   So sorry for the long delay. I didn't forget a...   Nov 10 2005, 20:10
|- - Woodinville   QUOTE (duff @ Nov 10 2005, 11:10 AM)Woodinsid...   Nov 10 2005, 20:23
- - duff   I don't understand why you would want to maint...   Nov 11 2005, 00:41
|- - Woodinville   QUOTE (duff @ Nov 10 2005, 03:41 PM)Somehow, ...   Nov 11 2005, 21:52
|- - Pio2001   QUOTE (duff @ Nov 11 2005, 01:41 AM)The ABX p...   Nov 12 2005, 03:00
|- - Woodinville   QUOTE (Pio2001 @ Nov 11 2005, 06:00 PM)I can ...   Nov 12 2005, 09:33
|- - user   There aren't theoretical flaws with abx or DB ...   Nov 12 2005, 10:52
- - KikeG   Sorry, you still have provided no evidence that wh...   Nov 11 2005, 14:34
|- - Woodinville   QUOTE (KikeG @ Nov 11 2005, 05:34 AM)Sorry, y...   Nov 11 2005, 21:56
|- - Pio2001   QUOTE (Woodinville @ Nov 11 2005, 10:56 PM)Su...   Nov 12 2005, 02:32
- - duff   Let's back up a moment... There are at least ...   Nov 11 2005, 18:37
- - KikeG   About the first point, your degraded examples are ...   Nov 11 2005, 19:57
- - duff   QUOTE Well, if it's inaudible, there's a g...   Nov 11 2005, 21:19
- - Woodinville   QUOTE (duff @ Nov 11 2005, 12:19 PM)In other ...   Nov 11 2005, 21:59
4 Pages V   1 2 3 > » 


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 28th July 2014 - 15:27