IPB

Welcome Guest ( Log In | Register )

Why Live-vs-Recorded Listening Tests Don't Work
solive
post Jul 10 2010, 11:08
Post #1





Group: Members
Posts: 162
Joined: 21-February 04
From: Los Angeles
Member No.: 12173



Thomas Edison was probably the greatest stereo salesman that ever lived. He believed that "listeners will hear what you tell them to hear", and he was pretty successful convincing thousands of listeners that his 1910 Edison Diamond Disc Phonograph reproduced recordings that sounded identical to a live performance. His secret weapon was an elaborate live-versus-recorded demonstration that managed to convince people that his phonograph sounded a lot better than it really was.

Several times over the past 10 years, I have been asked by live-versus-recorded apologists why I don't do these types of the tests since they claim they are the only true valid measures of loudspeaker fidelity or accuracy. That is what prompted me to write about why I believe live-versus-recorded listening tests don't work, in this month's blog article

Cheers
Sean Olive
Audio Musings

This post has been edited by solive: Jul 10 2010, 11:30


--------------------
Sean Olive
[url="http://seanolive.com"]Audio Musings[/url]
Go to the top of the page
+Quote Post
 
Start new topic
Replies
kdo
post Jul 10 2010, 16:53
Post #2





Group: Members (Donating)
Posts: 304
Joined: 18-April 02
From: Russia
Member No.: 1812



I have a couple of questions.

QUOTE
Live and Recorded Performances Must Be Identical

For live-versus-recorded tests to be valid, the live and recorded performance should be identical, having the same notes, intonation, tempo, dynamics, loudness, balance between instruments, and the same location and sense of space of the instruments. Otherwise, there are extraneous cues that allow listeners to readily identify the live and recorded performances. Midi-controlled instruments (e.g. player pianos) are but one example of how this problem could be resolved.


Would it be possible to design a valid test with the opposite approach? That is, instead of trying to reproduce a single performance identically, could we use various different performances and recordings every time?

I'm thinking of such scenario: suppose we need a test with 20 trials. Take 20 different singers with different voices, make 20 recordings. And then let some 20 more singers (again, all different voices) perform during the test, a sort of A/B test.
This way, I'm thinking, the singers don't even have to perform the same piece of music. It could be different music material every trial/performance.

Would it be possible to gather any statistically significant result from such a test?



And my 2nd question: can we consider our everyday practice of enjoying recorded music as the "ultimate" proof that such recordings are indeed capable of creating an illusion of live performance?
After several decades of such practical experience all across the globe, perhaps we already have enough evidence to draw some statistically valid conclusions? or still not?
I mean, okay, measuring the accuracy of a particular loudspeaker is one thing, but can't we say anything definitive of the technology in general?

This post has been edited by kdo: Jul 10 2010, 16:55
Go to the top of the page
+Quote Post
kdo
post Jul 11 2010, 16:25
Post #3





Group: Members (Donating)
Posts: 304
Joined: 18-April 02
From: Russia
Member No.: 1812



QUOTE (kdo @ Jul 10 2010, 17:53) *
Would it be possible to design a valid test with the opposite approach? That is, instead of trying to reproduce a single performance identically, could we use various different performances and recordings every time?


A quick follow-up on my first question.

I did some googling and found that I was actually thinking of a kind of test called "Randomized controlled trial" (RCT).
The "explanatory" type of RCT with "parallel-group" design and "allocation concealment", in particular. The goal of such RCT is to test the 'efficacy' of a treatment or medicine given to a group of patients.

So, here goes my analogy:
* 'efficacy' = ability to create illusion of a live performance.
* Participants (patients) are the singers/performers.
* Half of the participants are allocated to receive the 'treatment' (record and playback via loudspeakers),
and the other half is allocated to receive no 'treatment' (perform live).

The problem, I guess, is that it might be not quite properly triple-blind, since our 'participants' (singers) know which 'treatment' they are receiving, obviously.

But maybe this bias could be eliminated, too: let's make all singers perform live, but let some of them ("control group") perform before dummy-listeners in one room, and the other group perform before the actual audience in another room. Or something like that. Mix them and confuse them. smile.gif
Then the singers wouldn't know which of their performances would actually count, and the test becomes fully triple-blind, I hope. rolleyes.gif

So, on the surface it seems that it should be possible to eliminate the need for identical stimuli in a live-vs-recorded test - by using RCT method.

Any thoughts? Anybody? unsure.gif

Go to the top of the page
+Quote Post
solive
post Jul 13 2010, 00:43
Post #4





Group: Members
Posts: 162
Joined: 21-February 04
From: Los Angeles
Member No.: 12173



QUOTE (kdo @ Jul 11 2010, 08:25) *
QUOTE (kdo @ Jul 10 2010, 17:53) *
Would it be possible to design a valid test with the opposite approach? That is, instead of trying to reproduce a single performance identically, could we use various different performances and recordings every time?


A quick follow-up on my first question.

I did some googling and found that I was actually thinking of a kind of test called "Randomized controlled trial" (RCT).
The "explanatory" type of RCT with "parallel-group" design and "allocation concealment", in particular. The goal of such RCT is to test the 'efficacy' of a treatment or medicine given to a group of patients.

So, here goes my analogy:
* 'efficacy' = ability to create illusion of a live performance.
* Participants (patients) are the singers/performers.
* Half of the participants are allocated to receive the 'treatment' (record and playback via loudspeakers),
and the other half is allocated to receive no 'treatment' (perform live).

The problem, I guess, is that it might be not quite properly triple-blind, since our 'participants' (singers) know which 'treatment' they are receiving, obviously.

But maybe this bias could be eliminated, too: let's make all singers perform live, but let some of them ("control group") perform before dummy-listeners in one room, and the other group perform before the actual audience in another room. Or something like that. Mix them and confuse them. smile.gif
Then the singers wouldn't know which of their performances would actually count, and the test becomes fully triple-blind, I hope. rolleyes.gif

So, on the surface it seems that it should be possible to eliminate the need for identical stimuli in a live-vs-recorded test - by using RCT method.

Any thoughts? Anybody? unsure.gif


The participants in a listening test are normally the listeners - not the singers/performers, who act as one of the stimuli.

It is the listeners who decide whether or not the reproduction of the recording is similar to the live performance -- not the performers. I think you have misunderstood the original intent of the live-vs-recorded test.



--------------------
Sean Olive
[url="http://seanolive.com"]Audio Musings[/url]
Go to the top of the page
+Quote Post
Arnold B. Kruege...
post Jul 13 2010, 20:26
Post #5





Group: Members
Posts: 3797
Joined: 29-October 08
From: USA, 48236
Member No.: 61311



QUOTE (solive @ Jul 12 2010, 19:43) *
It is the listeners who decide whether or not the reproduction of the recording is similar to the live performance -- not the performers. I think you have misunderstood the original intent of the live-vs-recorded test.


An important point. While some performers have some sense of what their music sounds like, the audience knows far better what their music sounds like to the audience. That only makes common sense.

There's a reason why the preferred location for the mixer of a live performance is generally near the middle of the audience, and not the middle of the performers! ;-)
Go to the top of the page
+Quote Post

Posts in this topic
- solive   Why Live-vs-Recorded Listening Tests Don't Work   Jul 10 2010, 11:08
- - kdo   I have a couple of questions. QUOTE Live and Reco...   Jul 10 2010, 16:53
|- - kdo   QUOTE (kdo @ Jul 10 2010, 17:53) Would it...   Jul 11 2010, 16:25
||- - solive   QUOTE (kdo @ Jul 11 2010, 08:25) QUOTE (k...   Jul 13 2010, 00:43
|||- - kdo   QUOTE (solive @ Jul 13 2010, 01:43) The p...   Jul 13 2010, 01:02
||||- - solive   QUOTE (kdo @ Jul 12 2010, 17:02) QUOTE (s...   Jul 13 2010, 04:09
|||- - Arnold B. Krueger   QUOTE (solive @ Jul 12 2010, 19:43) It is...   Jul 13 2010, 20:26
||- - solive   QUOTE (kdo @ Jul 11 2010, 08:25) QUOTE (k...   Jul 13 2010, 04:53
||- - kdo   QUOTE (solive @ Jul 13 2010, 05:09) Sorry...   Jul 13 2010, 20:10
|- - solive   QUOTE (kdo @ Jul 10 2010, 08:53) I have a...   Jul 12 2010, 06:46
|- - kdo   QUOTE (solive @ Jul 12 2010, 07:46) 1) I ...   Jul 12 2010, 21:25
|- - Arnold B. Krueger   QUOTE (solive @ Jul 12 2010, 01:46) 1) I ...   Jul 13 2010, 17:25
|- - zane9   QUOTE (Arnold B. Krueger @ Jul 13 2010, 11...   Jul 13 2010, 18:50
||- - Arnold B. Krueger   QUOTE (zane9 @ Jul 13 2010, 13:50) QUOTE ...   Jul 13 2010, 20:21
||- - kdo   I sense a big fat TOS-8 violation right here: QUOT...   Jul 14 2010, 02:19
||- - Ed Seedhouse   One form of "live vs. reccorded" test wi...   Jul 14 2010, 02:35
|||- - kdo   QUOTE (Ed Seedhouse @ Jul 14 2010, 03:35)...   Jul 14 2010, 02:57
||- - Arnold B. Krueger   QUOTE (kdo @ Jul 13 2010, 21:19) I sense ...   Jul 14 2010, 03:12
||- - kdo   QUOTE (Arnold B. Krueger @ Jul 14 2010, 04...   Jul 14 2010, 03:30
||- - Arnold B. Krueger   QUOTE (kdo @ Jul 13 2010, 22:30) QUOTE (A...   Jul 14 2010, 10:42
||- - kdo   QUOTE (Arnold B. Krueger @ Jul 14 2010, 11...   Jul 14 2010, 16:56
||- - Arnold B. Krueger   QUOTE (kdo @ Jul 14 2010, 11:56) QUOTE (A...   Jul 14 2010, 17:15
||- - kdo   QUOTE (Arnold B. Krueger @ Jul 14 2010, 18...   Jul 14 2010, 17:58
|- - 2Bdecided   QUOTE (Arnold B. Krueger @ Jul 13 2010, 17...   Jul 14 2010, 19:04
|- - analog scott   QUOTE (2Bdecided @ Jul 14 2010, 19:04) QU...   Jul 14 2010, 20:40
|- - Arnold B. Krueger   QUOTE (2Bdecided @ Jul 14 2010, 14:04) It...   Jul 16 2010, 07:58
|- - 2Bdecided   QUOTE (Arnold B. Krueger @ Jul 16 2010, 07...   Jul 16 2010, 16:29
|- - Arnold B. Krueger   QUOTE (2Bdecided @ Jul 16 2010, 11:29) Cr...   Jul 16 2010, 23:59
- - analog scott   QUOTE (solive @ Jul 10 2010, 12:08) Thoma...   Jul 10 2010, 18:41
|- - solive   QUOTE (analog scott @ Jul 10 2010, 10:41)...   Jul 11 2010, 07:46
|- - analog scott   QUOTE (solive @ Jul 11 2010, 07:46) What ...   Jul 11 2010, 15:50
|- - googlebot   QUOTE (analog scott @ Jul 11 2010, 16:50)...   Jul 11 2010, 16:53
||- - analog scott   QUOTE (googlebot @ Jul 11 2010, 16:53) QU...   Jul 11 2010, 17:44
||- - aclo   QUOTE (googlebot @ Jul 11 2010, 17:53) Wh...   Jul 12 2010, 04:06
|- - solive   QUOTE (analog scott @ Jul 11 2010, 07:50)...   Jul 11 2010, 20:16
||- - analog scott   QUOTE (solive @ Jul 11 2010, 21:16) QUOTE...   Jul 11 2010, 20:33
||- - googlebot   QUOTE (solive @ Jul 11 2010, 21:16) The r...   Jul 11 2010, 20:33
|||- - solive   QUOTE (googlebot @ Jul 11 2010, 12:33) QU...   Jul 12 2010, 06:19
||- - 2Bdecided   QUOTE (solive @ Jul 11 2010, 20:16) The r...   Jul 14 2010, 18:54
|- - greynol   QUOTE (analog scott @ Jul 11 2010, 07:50)...   Jul 11 2010, 20:28
|- - Notat   QUOTE (greynol @ Jul 11 2010, 13:28) It...   Jul 11 2010, 23:06
- - greynol   Quite unfortunate if someone wants to garner somet...   Jul 11 2010, 23:19
- - greynol   Unless he can assure us that his opinion about the...   Jul 12 2010, 02:26
- - greynol   analogscott's post binned per TOS #2. Further...   Jul 12 2010, 03:09


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 23rd September 2014 - 16:41