IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
A Descriptive Evaluation Methodology for Consumer Audio Equipment
krabapple
post Sep 30 2010, 22:32
Post #1





Group: Members
Posts: 2184
Joined: 18-December 03
Member No.: 10538



http://onlinelibrary.wiley.com/doi/10.1111...10.00306.x/full


A DESCRIPTIVE EVALUATION METHODOLOGY FOR CONSUMER AUDIO EQUIPMENT
RAIFE F. SMITH II

QUOTE
ABSTRACT

A pilot study was conducted in order to investigate the application of descriptive analysis techniques to the evaluation of consumer audio equipment. A global seven-step methodology for the descriptive evaluation of the sound characteristics of consumer audio equipment was developed. In contrast to the commonly used forced choice discrimination methods for audio equipment evaluation, the proposed methodology enhances, rather than compromises, a subject's sensory input. A case study employing a minimally trained subject demonstrated the applicability of sensory evaluation techniques to sound. There has been some resistance in the audio community to the adoption of sensory science evaluation methods because such methods are erroneously thought to apply only to products that affect the senses of touch, taste, smell and sight.



discuss.


btw, Dr. Smith is also the author of this rather curious article in 'Affordable$$Audio''


http://www.affordableaudio.org/aa2010-09.pdf
Go to the top of the page
+Quote Post
pdq
post Sep 30 2010, 23:27
Post #2





Group: Members
Posts: 3374
Joined: 1-September 05
From: SE Pennsylvania
Member No.: 24233



It appears that his main justification for rejecting ABX testing is that it was once used to test audio quality of telephones. laugh.gif
Go to the top of the page
+Quote Post
Arnold B. Kruege...
post Oct 1 2010, 16:51
Post #3





Group: Members
Posts: 3650
Joined: 29-October 08
From: USA, 48236
Member No.: 61311



QUOTE (pdq @ Sep 30 2010, 18:27) *
It appears that his main justification for rejecting ABX testing is that it was once used to test audio quality of telephones. laugh.gif


He says:

"Blind audio testing, which includes visually obscuring all or part of the sound stage, rapid switching of musical selections and off-axis and group
seating, impairs the listener's ability to localize sounds (seeing), to internalize and evaluate aural cues (hearing) and to
receive correct stereophonic tactile information (touching). Any stereophonic audio system testing methodology which
compromises and hinders the processes of human sensory perception will result in consistently inaccurate and often
absurd results."

This is one of the most confused paragraphs I've ever read about listening tests. It is a total misrepresentation of the true facts.

Since when does ABX test include "visually obscuring all or part of the sound stage"? I guess the author thinks that blind testing necessarily means wearing blindfolds, poking one eyes out, or darkering the room.

As has been pointed out many times, rapid switching of musical selections is not part of ABX testing. What may be rapdily switched is which presentation of the musical lsitening selections is being listened to at any particular time. Or not.

Since ABX tests are frequently done with just one listener, and since headphone listening is commonly done during ABX tests, the comments about off-axix and group seating are equally irreleevant to the true factys.

This paper is bascially a straw man argument. The apparently athor makes up his own ideas about what a bad listening test might be like, and then names them "ABX". The artile looks to me like baldfaced libel. I'd be worried if the author had any influence.
Go to the top of the page
+Quote Post
analog scott
post Oct 2 2010, 01:12
Post #4





Group: Members
Posts: 332
Joined: 26-July 09
Member No.: 71796



QUOTE (Arnold B. Krueger @ Oct 1 2010, 17:51) *
QUOTE (pdq @ Sep 30 2010, 18:27) *
It appears that his main justification for rejecting ABX testing is that it was once used to test audio quality of telephones. laugh.gif


He says:

"Blind audio testing, which includes visually obscuring all or part of the sound stage, rapid switching of musical selections and off-axis and group
seating, impairs the listener's ability to localize sounds (seeing), to internalize and evaluate aural cues (hearing) and to
receive correct stereophonic tactile information (touching). Any stereophonic audio system testing methodology which
compromises and hinders the processes of human sensory perception will result in consistently inaccurate and often
absurd results."

This is one of the most confused paragraphs I've ever read about listening tests. It is a total misrepresentation of the true facts.

Since when does ABX test include "visually obscuring all or part of the sound stage"? I guess the author thinks that blind testing necessarily means wearing blindfolds, poking one eyes out, or darkering the room.

As has been pointed out many times, rapid switching of musical selections is not part of ABX testing. What may be rapdily switched is which presentation of the musical lsitening selections is being listened to at any particular time. Or not.

Since ABX tests are frequently done with just one listener, and since headphone listening is commonly done during ABX tests, the comments about off-axix and group seating are equally irreleevant to the true factys.

This paper is bascially a straw man argument. The apparently athor makes up his own ideas about what a bad listening test might be like, and then names them "ABX". The artile looks to me like baldfaced libel. I'd be worried if the author had any influence.


That is a big a$$ compound sentence but it is actually true if you break it down for what it is and only what it is. He is describing a specific subset of "blind audio testing." That subset includes a visual obscuring of the "sound stage" (not sure why this would be an issue) rapid switching of musical selctions, and "off-axis and group seating." The last part is an issue. You will mess up the imaging of stereo playback *if* you do test that way.

If the author makes any broad inferences about blind testing from this specific subset conditions then that would be problematic to say the least.
Go to the top of the page
+Quote Post
krabapple
post Oct 2 2010, 05:58
Post #5





Group: Members
Posts: 2184
Joined: 18-December 03
Member No.: 10538



QUOTE (analog scott @ Oct 1 2010, 20:12) *
That is a big a$$ compound sentence but it is actually true ihttp://www.hydrogenaudio.org/forums/index.php?act=Post&CODE=02&f=40&t=83979&qpid=725107f you break it down for what it is and only what it is. He is describing a specific subset of "blind audio testing." That subset includes a visual obscuring of the "sound stage" (not sure why this would be an issue) rapid switching of musical selctions, and "off-axis and group seating." The last part is an issue. You will mess up the imaging of stereo playback *if* you do test that way.

If the author makes any broad inferences about blind testing from this specific subset conditions then that would be problematic to say the least.


'To say the least'? Read the article before commenting please, Scott. Obviously you have not. Go ahead, it's a free download.

The paragraph quoted is the *concluding paragraph* of the article. If he was writing objectively, Dr. Smith would have written something like, 'blind tests have been conducted where in addition to being blind, ....", clearly separating the conditions which are intrinsic to a blind test, from those that are not. As written, he appears to be trying to tar audio blind testing itself or 'at the very least' obfuscating the issue by the use of the ambiguous term 'include'. This is on top of multiple risible claims elsewhere in the article, like "According to Lipshitz, Vanderkooy and Clark, It is better to blindfold or otherwise visually handicap music listeners".

And btw, your reflexive need to undermine anything Arny writes remains as pathetic as it is predictable. Please be aware that some of us here know of the long history between you two.

This post has been edited by krabapple: Oct 2 2010, 06:04
Go to the top of the page
+Quote Post
analog scott
post Oct 2 2010, 07:33
Post #6





Group: Members
Posts: 332
Joined: 26-July 09
Member No.: 71796



QUOTE (krabapple @ Oct 2 2010, 06:58) *
QUOTE (analog scott @ Oct 1 2010, 20:12) *
That is a big a$$ compound sentence but it is actually true ihttp://www.hydrogenaudio.org/forums/index.php?act=Post&CODE=02&f=40&t=83979&qpid=725107f you break it down for what it is and only what it is. He is describing a specific subset of "blind audio testing." That subset includes a visual obscuring of the "sound stage" (not sure why this would be an issue) rapid switching of musical selctions, and "off-axis and group seating." The last part is an issue. You will mess up the imaging of stereo playback *if* you do test that way.

If the author makes any broad inferences about blind testing from this specific subset conditions then that would be problematic to say the least.


'To say the least'? Read the article before commenting please, Scott. Obviously you have not. Go ahead, it's a free download.

The paragraph quoted is the *concluding paragraph* of the article. If he was writing objectively, Dr. Smith would have written something like, 'blind tests have been conducted where in addition to being blind, ....", clearly separating the conditions which are intrinsic to a blind test, from those that are not. As written, he appears to be trying to tar audio blind testing itself or 'at the very least' obfuscating the issue by the use of the ambiguous term 'include'. This is on top of multiple risible claims elsewhere in the article, like "According to Lipshitz, Vanderkooy and Clark, It is better to blindfold or otherwise visually handicap music listeners".

And btw, your reflexive need to undermine anything Arny writes remains as pathetic as it is predictable. Please be aware that some of us here know of the long history between you two.


Yes Steve, "to say the least." Do you understand what that means? It was a punch line, a sarcastic insult in the form of an understatement. Arny quoted the specific sentence and felt it was " one of the most confused paragraphs I've ever read about listening tests. It is a total misrepresentation of the true facts." So my comments were limited to that paragraph. Well, actually it was just a sentence. I don't see my comments as "undermining" Arny. I was trying to have some fun with a poorly written sentence that was painfully convoluted. I doubt that Arny would disagree that blind testing in stereo with "off axis and group seating" is a problem. Obviously the conditions listed in that "confused" sentence/paragraph were so specific and pointed as to have little or no meaning in the big picture. Oh yeah, I kinda said that didn't I?

Dude, lighten up. I was making fun of the sentence/paragraph. Oh well, once one has to explain the joke the joke dies.

Go to the top of the page
+Quote Post
Arnold B. Kruege...
post Oct 3 2010, 13:03
Post #7





Group: Members
Posts: 3650
Joined: 29-October 08
From: USA, 48236
Member No.: 61311



QUOTE (analog scott @ Oct 1 2010, 20:12) *
QUOTE (Arnold B. Krueger @ Oct 1 2010, 17:51) *
QUOTE (pdq @ Sep 30 2010, 18:27) *
It appears that his main justification for rejecting ABX testing is that it was once used to test audio quality of telephones. laugh.gif


He says:

"Blind audio testing, which includes visually obscuring all or part of the sound stage, rapid switching of musical selections and off-axis and group
seating, impairs the listener's ability to localize sounds (seeing), to internalize and evaluate aural cues (hearing) and to
receive correct stereophonic tactile information (touching). Any stereophonic audio system testing methodology which
compromises and hinders the processes of human sensory perception will result in consistently inaccurate and often
absurd results."

This is one of the most confused paragraphs I've ever read about listening tests. It is a total misrepresentation of the true facts.

Since when does ABX test include "visually obscuring all or part of the sound stage"? I guess the author thinks that blind testing necessarily means wearing blindfolds, poking one eyes out, or darkering the room.

As has been pointed out many times, rapid switching of musical selections is not part of ABX testing. What may be rapdily switched is which presentation of the musical lsitening selections is being listened to at any particular time. Or not.

Since ABX tests are frequently done with just one listener, and since headphone listening is commonly done during ABX tests, the comments about off-axix and group seating are equally irreleevant to the true factys.

This paper is bascially a straw man argument. The apparently athor makes up his own ideas about what a bad listening test might be like, and then names them "ABX". The artile looks to me like baldfaced libel. I'd be worried if the author had any influence.


That is a big a$$ compound sentence but it is actually true if you break it down for what it is and only what it is. He is describing a specific subset of "blind audio testing." That subset includes a visual obscuring of the "sound stage" (not sure why this would be an issue) rapid switching of musical selctions, and "off-axis and group seating." The last part is an issue. You will mess up the imaging of stereo playback *if* you do test that way.

If the author makes any broad inferences about blind testing from this specific subset conditions then that would be problematic to say the least.


I see nothing in the entire article that suggests that the author says hs is adressing just a subset of blind listening tests. If you can find it, quote it. If you can't, we'll chalk it up as just another one of your many hihgly flawed attempts to undermine listening tests that fail to justify your extensive and expensive purchases of non-functional and poorly-functioning boutique audio gear.
Go to the top of the page
+Quote Post
googlebot
post Oct 3 2010, 13:42
Post #8





Group: Members
Posts: 698
Joined: 6-March 10
Member No.: 78779



QUOTE (analog scott @ Oct 2 2010, 08:33) *
Dude, lighten up. I was making fun of the sentence/paragraph. Oh well, once one has to explain the joke the joke dies.


The dominant impression was one of contentious spirit and prevarication for me. You might add a bold-faced "Bazinga!" next time, so that the simple-minded among us also get the humor. wink.gif
Go to the top of the page
+Quote Post
analog scott
post Oct 3 2010, 15:41
Post #9





Group: Members
Posts: 332
Joined: 26-July 09
Member No.: 71796



QUOTE (googlebot @ Oct 3 2010, 13:42) *
QUOTE (analog scott @ Oct 2 2010, 08:33) *
Dude, lighten up. I was making fun of the sentence/paragraph. Oh well, once one has to explain the joke the joke dies.


The dominant impression was one of contentious spirit and prevarication for me. You might add a bold-faced "Bazinga!" next time, so that the simple-minded among us also get the humor. wink.gif



Thanks for the tip. Did not know that Bold face was internet code for dry sarcasm or something like that. Sorry for any misunderstandings. It was meant as a back handed compliment to the author of the article. Kind of like the old Monty Python bit, "what have the Romans ever done for us. besides the...blah blah blah...." only in reverse. He's right about DBTs if....blah blah blah.... The absurdity of the conditions being the irony. Well anyway...like I said... once ya gotta explain the joke the funny dies. crying.gif
Go to the top of the page
+Quote Post
Speedskater
post Oct 3 2010, 16:13
Post #10





Group: Members
Posts: 137
Joined: 12-December 07
From: Cleveland, OH
Member No.: 49500



I believe that Mr. Smith often posts on the Polk Audio Forum as DarqueKnight. He started a review of audio grade fuses which he liked, but interestingly he referred to himself as a "jaded skeptic".


--------------------
Kevin
Go to the top of the page
+Quote Post
Speedskater
post Oct 3 2010, 16:37
Post #11





Group: Members
Posts: 137
Joined: 12-December 07
From: Cleveland, OH
Member No.: 49500



More on Dr. Smith and some of his topics in the Polk Audio Forum:

Southern Univerisity and A & M College
Dr. Raife F. Smith II, Associate Professor, Electrical Engineering
Ph.D. Electrical Engineering, Tulane University

Dr. Smith's teaching and research interests are in the areas of Communications Signals and Systems, Broadband Telecommunications Network Design and Optimization, and Stochastic Modeling. Dr. Smith's consulting interests are in enterprise network and public switched network modeling, design and optimization.

Notice Of Sensory Science Journal Publication
http://www.polkaudio.com/forums/showthread.php?t=103417

A Historical Overview of Stereophonic Blind Testing
http://www.polkaudio.com/forums/showthread.php?t=104973

A Survey Of Early Stereophonic System Subjective Evaluation
http://www.polkaudio.com/forums/showthread.php?t=104701

Observations Of A "Good" Cable Test
http://www.polkaudio.com/forums/showthread.php?t=104101

Studies On Residential Power Line Noise - Part 3- PS Audio Power Plant Premier
http://www.polkaudio.com/forums/showthread.php?t=71545

What Is "High End" Audio?
http://www.polkaudio.com/forums/showthread.php?t=95998

JuiceCyclone Power Cable Conditioner
http://www.polkaudio.com/forums/showthread.php?t=89349

This post has been edited by Speedskater: Oct 3 2010, 16:37


--------------------
Kevin
Go to the top of the page
+Quote Post
krabapple
post Oct 4 2010, 00:31
Post #12





Group: Members
Posts: 2184
Joined: 18-December 03
Member No.: 10538



I pulled this gem from one of those polkaudio trainwrecks:

QUOTE
The fundamental flaw in applying ABX tests to audio evaluation is in that people participating in ABX tests are searching for differences between equipment rather than conducting a performance evaluation against standard performance criteria (metrics).

In other words, ABX is fundamentally flawed because it compares gear to gear rather than comparing gear to performance standards. This is ludicrous and demonstrates an ignorance of the purpose of stereophonic music reproduction. This seems to be a concept that the "objectivists" and "scientists" cannot, or will not, grasp
.


It's saddening to read because he employs academic sensory testing jargon, and mixes 'real' science with tendentious crap.
E.g., this amazing jumble of fact and fiction:

http://www.polkaudio.com/forums/showpost.p...mp;postcount=53

QUOTE
1. An ABX (duo-trio balanced reference) test generally requires a subject population of at least 16 persons. Optimum subject population is at least 32 or more persons.

2. An ABX (duo-trio balanced reference) test which employs a subject size of less than 28 persons generates high rates of beta error (false negatives or "no differences between samples") in the results.

3. An ABX (duo-trio balanced reference) test must compare samples which are unknown (unfamiliar) to the test subjects.

4. An ABX (duo-trio balanced reference) test must use untrained test subjects.


This post has been edited by krabapple: Oct 4 2010, 00:44
Go to the top of the page
+Quote Post
Arnold B. Kruege...
post Oct 4 2010, 14:18
Post #13





Group: Members
Posts: 3650
Joined: 29-October 08
From: USA, 48236
Member No.: 61311



QUOTE (krabapple @ Oct 3 2010, 19:31) *
I pulled this gem from one of those polkaudio trainwrecks:

QUOTE
The fundamental flaw in applying ABX tests to audio evaluation is in that people participating in ABX tests are searching for differences between equipment rather than conducting a performance evaluation against standard performance criteria (metrics).

In other words, ABX is fundamentally flawed because it compares gear to gear rather than comparing gear to performance standards. This is ludicrous and demonstrates an ignorance of the purpose of stereophonic music reproduction. This seems to be a concept that the "objectivists" and "scientists" cannot, or will not, grasp
.


It's saddening to read because he employs academic sensory testing jargon, and mixes 'real' science with tendentious crap.
E.g., this amazing jumble of fact and fiction:

http://www.polkaudio.com/forums/showpost.p...mp;postcount=53

QUOTE
1. An ABX (duo-trio balanced reference) test generally requires a subject population of at least 16 persons. Optimum subject population is at least 32 or more persons.

2. An ABX (duo-trio balanced reference) test which employs a subject size of less than 28 persons generates high rates of beta error (false negatives or "no differences between samples") in the results.

3. An ABX (duo-trio balanced reference) test must compare samples which are unknown (unfamiliar) to the test subjects.

4. An ABX (duo-trio balanced reference) test must use untrained test subjects.




Amazing stuff. I wonder if he will be sucessful at marketing himself as a staff writer for one of the high end ragazines?
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 30th July 2014 - 13:12