Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Why Live-vs-Recorded Listening Tests Don't Work (Read 28680 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Why Live-vs-Recorded Listening Tests Don't Work

Thomas Edison was probably the greatest stereo salesman that ever lived. He believed that "listeners will hear what you tell them to hear", and he was pretty successful convincing thousands of listeners that his 1910 Edison Diamond Disc Phonograph reproduced recordings that sounded identical to a live performance.  His secret weapon was an elaborate live-versus-recorded demonstration that managed to convince  people that his phonograph sounded a lot better than it really was.

Several times over the past 10 years, I have been asked by live-versus-recorded apologists why I don't do these types of the tests since they claim they are the only true valid measures of loudspeaker fidelity or accuracy.  That is what prompted me to write about why I believe live-versus-recorded listening tests don't  work, in this month's blog article

Cheers
Sean Olive
Audio Musings

Why Live-vs-Recorded Listening Tests Don't Work

Reply #1
I have a couple of questions.

Quote
Live and Recorded Performances Must Be Identical

For live-versus-recorded tests to be valid, the live and recorded performance should be identical, having the same notes, intonation, tempo, dynamics, loudness, balance between instruments, and the same location and sense of space of the instruments. Otherwise, there are extraneous cues that allow listeners to readily identify the live and recorded performances. Midi-controlled instruments (e.g. player pianos) are but one example of how this problem could be resolved.


Would it be possible to design a valid test with the opposite approach? That is, instead of trying to reproduce a single performance identically, could we use various different performances and recordings every time?

I'm thinking of such scenario: suppose we need a test with 20 trials. Take 20 different singers with different voices, make 20 recordings. And then let some 20 more singers (again, all different voices) perform during the test, a sort of A/B test.
This way, I'm thinking, the singers don't even have to perform the same piece of music. It could be different music material every trial/performance.

Would it be possible to gather any statistically significant result from such a test?



And my 2nd question: can we consider our everyday practice of enjoying recorded music as the "ultimate" proof that such recordings are indeed capable of creating an illusion of live performance?
After several decades of such practical experience all across the globe, perhaps we already have enough evidence to draw some statistically valid conclusions? or still not?
I mean, okay, measuring the accuracy of a particular loudspeaker is one thing, but can't we say anything definitive of the technology in general?

Why Live-vs-Recorded Listening Tests Don't Work

Reply #2
Thomas Edison was probably the greatest stereo salesman that ever lived. He believed that "listeners will hear what you tell them to hear", and he was pretty successful convincing thousands of listeners that his 1910 Edison Diamond Disc Phonograph reproduced recordings that sounded identical to a live performance.  His secret weapon was an elaborate live-versus-recorded demonstration that managed to convince  people that his phonograph sounded a lot better than it really was.

Several times over the past 10 years, I have been asked by live-versus-recorded apologists why I don't do these types of the tests since they claim they are the only true valid measures of loudspeaker fidelity or accuracy.  That is what prompted me to write about why I believe live-versus-recorded listening tests don't  work, in this month's blog article

Cheers
Sean Olive
Audio Musings


Very nice article Sean. I think you make a valid point that one can't use live music as a means of judging loudspeakers per se. But it seems to me that one can use live music as a reference to judge a recording and playback system in it's totallity.

Why Live-vs-Recorded Listening Tests Don't Work

Reply #3
Thomas Edison was probably the greatest stereo salesman that ever lived. He believed that "listeners will hear what you tell them to hear", and he was pretty successful convincing thousands of listeners that his 1910 Edison Diamond Disc Phonograph reproduced recordings that sounded identical to a live performance.  His secret weapon was an elaborate live-versus-recorded demonstration that managed to convince  people that his phonograph sounded a lot better than it really was.

Several times over the past 10 years, I have been asked by live-versus-recorded apologists why I don't do these types of the tests since they claim they are the only true valid measures of loudspeaker fidelity or accuracy.  That is what prompted me to write about why I believe live-versus-recorded listening tests don't  work, in this month's blog article

Cheers
Sean Olive
Audio Musings


Very nice article Sean. I think you make a valid point that one can't use live music as a means of judging loudspeakers per se. But it seems to me that one can use live music as a reference to judge a recording and playback system in it's totallity.


Thanks. Yes, you can certainly judge the accuracy of the entire recording/playback chain to a live performance but the test makes it difficult to know which component is responsible for the artifacts: the recording, the loudspeakers, or both.

In my view, the closest you can come to recreating a live performance experience is a binaural recording/room scan with a head-tracking headphone-based auditory display. Stereo just doesn't cut it, but multichannel gets a lot closer if the recording is done well.  And there are still challenges controlling all the nuisance variables.

What about recordings that are not intended to sound like a live performance?  That would include about 95% of all recording made today. How do we judge the accuracy of those? Of course, you know the answer to that question: you define the performance of the loudspeakers and their interaction with the room acoustics where the art (the recording)  was created, and  simply replicate the playback system in the consumer space. Science in the service of art -- Not a  popular concept among the live-versus-recording apologists I've met.




Cheers
Sean Olive
Audio Musings

Why Live-vs-Recorded Listening Tests Don't Work

Reply #4
What about recordings that are not intended to sound like a live performance?  That would include about 95% of all recording made today. How do we judge the accuracy of those?



I can't say that it is an issue to me. But if one is going to ask the question then one has to define the reference. What is the reference for a studio recording if one is seeking an "accurate" reproduction of it? Unlike a recording of an original live performance a studio recording in and of itself has no intrinsic original sound.


Of course, you know the answer to that question: you define the performance of the loudspeakers and their interaction with the room acoustics where the art (the recording)  was created, and  simply replicate the playback system in the consumer space. Science in the service of art -- Not a  popular concept among the live-versus-recording apologists I've met.




Cheers
Sean Olive
Audio Musings



Yep and no thanks.

But this points to a bigger question in audio. Why seek accuracy? Why use live music or the sound originally heard in the control room as a reference? As an audiophile I think in this quest for accuracy the forest has been lost for the trees. For me live music is really a benchmark more than a literal rigid "reference." The most beautiful sounding music I have heard has come from live acoustic music be it a symphony orchestra at Disney Hall or that magical concert I went to in a church in Soweto with Ladysmith Black Mambazo (maybe the most beautiful thing I have ever heard) or any number of other magical moments I have experienced with live acoustic jazz or folk etc. These experiecneces for me have been the pinnicles of aesthetic beauty in sound. So with recording and playback of live music it consistantly seems to work that the closer you get to the sounds one hears with live music (with all the qualifiers) the better the playback tends to be. Heck if some day I hear something on a hifi that simply sounds better than anything I have ever heard live...that becomes the new benchmark for me. If live music didn't set the benchmark there would be no point in using it as a reference and no point in trying to accurately recreate those sounds.

Now with studio recordings....IMO the control rooms don't set such a lofty benchmark. IOW IMO one can do much better than "accurate" with those recordings. So I see no point in seeking accuracy with such recordings.


Why Live-vs-Recorded Listening Tests Don't Work

Reply #5
Would it be possible to design a valid test with the opposite approach? That is, instead of trying to reproduce a single performance identically, could we use various different performances and recordings every time?


A quick follow-up on my first question.

I did some googling and found that I was actually thinking of a kind of test called "Randomized controlled trial" (RCT).
The "explanatory" type of RCT with "parallel-group" design and "allocation concealment", in particular. The goal of such RCT is to test the 'efficacy' of a treatment or medicine given to a group of patients.

So, here goes my analogy:
* 'efficacy' = ability to create illusion of a live performance.
* Participants (patients) are the singers/performers.
* Half of the participants are allocated to receive the 'treatment' (record and playback via loudspeakers),
and the other half is allocated to receive no 'treatment' (perform live).

The problem, I guess, is that it might be not quite properly triple-blind, since our 'participants' (singers) know which 'treatment' they are receiving, obviously.

But maybe this bias could be eliminated, too: let's make all singers perform live, but let some of them ("control group") perform before dummy-listeners in one room, and the other group perform before the actual audience in another room. Or something like that. Mix them and confuse them.
Then the singers wouldn't know which of their performances would actually count, and the test becomes fully triple-blind, I hope. 

So, on the surface it seems that it should be possible to eliminate the need for identical stimuli in a live-vs-recorded test - by using RCT method.

Any thoughts? Anybody? 


Why Live-vs-Recorded Listening Tests Don't Work

Reply #6
Now with studio recordings....IMO the control rooms don't set such a lofty benchmark. IOW IMO one can do much better than "accurate" with those recordings. So I see no point in seeking accuracy with such recordings.


Your problem is just a narrow understanding of the word "accurate". No one wants an accurate reproduction of a low resonance studio room on a final record (and no one has claimed that, either). Accurate studio equipment in this context just means it does not add additional artifacts by itself. A professional skilled in the art will process it with the tools of his choice and create something not resembling anything close to a studio chamber. So your whole excursion is kind of pointless.

Even most live recordings aren't mixed down with the goal of accurate sonic reproduction of a singular listening position anymore. Since records are generally played back in rooms with walls and a finite number of speakers, that's impossible anyway. So people searching that could be disappointed.

While going to a concert in a good hall can be an exceptional experience, I have experienced the most detail in classical music from great records. If you want every detail, IMHO, the best records have surpassed live performances (which doesn't necessarily make it the better experience) for quite some time now. But experience is hard to quantify anyway, so why don't enjoy the best of both worlds?

Why Live-vs-Recorded Listening Tests Don't Work

Reply #7
Now with studio recordings....IMO the control rooms don't set such a lofty benchmark. IOW IMO one can do much better than "accurate" with those recordings. So I see no point in seeking accuracy with such recordings.


Your problem is just a narrow understanding of the word "accurate".



1. I don't have a problem, I'm doin just fine thank you.
2. A "narrow understanding" of the word accurate? Sorry not interested in semantical arguments. I know what "accurate" means.

You might want to take this argument up with Sean as well since he was the one who said "What about recordings that are not intended to sound like a live performance? That would include about 95% of all recording made today. How do we judge the accuracy of those? Of course, you know the answer to that question: you define the performance of the loudspeakers and their interaction with the room acoustics where the art (the recording) was created, and simply replicate the playback system in the consumer space." I guess Sean suffers from the same "narrow understanding" of the word "accurate." 

No one wants an accurate reproduction of a low resonance studio room on a final record (and no one has claimed that, either).



Well actually people have claimed that they do want that. I have seen it with my own eyes on other forums. Maybe not such a good idea to speak for other unnnamed people.


Accurate studio equipment in this context just means it does not add additional artifacts by itself. A professional skilled in the art will process it with the tools of his choice and create something not resembling anything close to a studio chamber. So your whole excursion is kind of pointless.


Who said anything about "accurate studio equipment?" Again you might want to take this argument up with Sean as well. Sean's words "What about recordings that are not intended to sound like a live performance? That would include about 95% of all recording made today. How do we judge the accuracy of those? Of course, you know the answer to that question: you define the performance of the loudspeakers and their interaction with the room acoustics where the art (the recording) was created, and simply replicate the playback system in the consumer space." In case it isn't clear the "loudspeakers and their interatcion with the room acoustics where the art (the recording) was created" is the control room.

By the way, it's not *my* excursion. I think I made my position quite clear when it comes to seeking "accuracy" with studio recordings. I believe I said "no thanks." all I did was agree with Sean that if one wants an aural reference for studio recordings in the persuit of accuracy one is ultimately left with what was heard in the control room. I suppose one can take it a step further and claim what was heard in the room when final approval was given for the commerical release is the final reference but....





While going to a concert in a good hall can be an exceptional experience, I have experienced the most detail in classical music from great records. If you want every detail, IMHO, the best records have surpassed live performances (which doesn't necessarily make it the better experience) for quite some time now.




I want detail to be detail. I'm not keen on exaggerated detail. I have seen forensic photographs that show more detail than one could ever see with the naked eye. But they are ugly pictures. Heck you can hear more detail just by jacking up the treble and compressing the signal. You will certainly get more detail from an acoustic instrument with a close miked recording in an anechoic chamber. None of that nasty reverb to obscure the sound of the instrument itself.  But for me, much like a forensic photograph, exaggerated detail beyond what we hear in live music tends towards the ugly. Not my thing.

But experience is hard to quantify anyway, so why don't enjoy the best of both worlds?




That is what I have been doing all along.

Why Live-vs-Recorded Listening Tests Don't Work

Reply #8


Quote
I can't say that it is an issue to me. But if one is going to ask the question then one has to define the reference. What is the reference for a studio recording if one is seeking an "accurate" reproduction of it? Unlike a recording of an original live performance a studio recording in and of itself has no intrinsic original sound.


The reference is what the artist heard in the studio plain and simple. That is the "live performance"


Quote
Yep and no thanks.

But this points to a bigger question in audio. Why seek accuracy? Why use live music or the sound originally heard in the control room as a reference? As an audiophile I think in this quest for accuracy the forest has been lost for the trees. For me live music is really a benchmark more than a literal rigid "reference." The most beautiful sounding music I have heard has come from live acoustic music be it a symphony orchestra at Disney Hall or that magical concert I went to in a church in Soweto with Ladysmith Black Mambazo (maybe the most beautiful thing I have ever heard) or any number of other magical moments I have experienced with live acoustic jazz or folk etc. These experiecneces for me have been the pinnicles of aesthetic beauty in sound. So with recording and playback of live music it consistantly seems to work that the closer you get to the sounds one hears with live music (with all the qualifiers) the better the playback tends to be. Heck if some day I hear something on a hifi that simply sounds better than anything I have ever heard live...that becomes the new benchmark for me. If live music didn't set the benchmark there would be no point in using it as a reference and no point in trying to accurately recreate those sounds.

Now with studio recordings....IMO the control rooms don't set such a lofty benchmark. IOW IMO one can do much better than "accurate" with those recordings. So I see no point in seeking accuracy with such recordings.


Wow, that's quite an admission that you don't care about accuracy in sound reproduction.  I would put that in your signature "Analog Scott - "I don't care about accuracy" since that would save a lot of people the hassle and time arguing with you -

But you raise a valid point. Certainly on the recording side, most engineers/producers are not seeking to accurately capture/reproduce the "live performance"  which makes the whole live-vs-recorded method rather moot.


Cheers
Sean Olive
Audio Musings

Why Live-vs-Recorded Listening Tests Don't Work

Reply #9
These experiecneces for me have been the pinnicles of aesthetic beauty in sound.

It's unfortunate that these anecdotal experiences are completely subjective and were likely tainted by things that had nothing to do with actual sound.

Why Live-vs-Recorded Listening Tests Don't Work

Reply #10


Quote
I can't say that it is an issue to me. But if one is going to ask the question then one has to define the reference. What is the reference for a studio recording if one is seeking an "accurate" reproduction of it? Unlike a recording of an original live performance a studio recording in and of itself has no intrinsic original sound.


The reference is what the artist heard in the studio plain and simple. That is the "live performance"


Quote
Yep and no thanks.

But this points to a bigger question in audio. Why seek accuracy? Why use live music or the sound originally heard in the control room as a reference? As an audiophile I think in this quest for accuracy the forest has been lost for the trees. For me live music is really a benchmark more than a literal rigid "reference." The most beautiful sounding music I have heard has come from live acoustic music be it a symphony orchestra at Disney Hall or that magical concert I went to in a church in Soweto with Ladysmith Black Mambazo (maybe the most beautiful thing I have ever heard) or any number of other magical moments I have experienced with live acoustic jazz or folk etc. These experiecneces for me have been the pinnicles of aesthetic beauty in sound. So with recording and playback of live music it consistantly seems to work that the closer you get to the sounds one hears with live music (with all the qualifiers) the better the playback tends to be. Heck if some day I hear something on a hifi that simply sounds better than anything I have ever heard live...that becomes the new benchmark for me. If live music didn't set the benchmark there would be no point in using it as a reference and no point in trying to accurately recreate those sounds.

Now with studio recordings....IMO the control rooms don't set such a lofty benchmark. IOW IMO one can do much better than "accurate" with those recordings. So I see no point in seeking accuracy with such recordings.


Wow, that's quite an admission that you don't care about accuracy in sound reproduction.  I would put that in your signature "Analog Scott - "I don't care about accuracy" since that would save a lot of people the hassle and time arguing with you -

But you raise a valid point. Certainly on the recording side, most engineers/producers are not seeking to accurately capture/reproduce the "live performance"  which makes the whole live-vs-recorded method rather moot.


Cheers
Sean Olive
Audio Musings



I certainly don't care about accuracy for the sake of accuracy. I care about it in so far as it serves the higher purpose of a better aesthetic experience. Some times it serves that purpose some times it does not. Here is a great example, the Led Zeppelin BBC sessions that were mastered with Jimmy Page overseeing it. He was as much as anybody "the artist." But he is in his sixties and almost certainly is suffering from serious hearing damage. The result...ear bleeding bright compressed sound. Should I want this sound as accurately reproduced as possible because it is accurate?

I don't care about accuracy. I care about aesthetic excellence.

Why Live-vs-Recorded Listening Tests Don't Work

Reply #11
The reference is what the artist heard in the studio plain and simple. That is the "live performance"


Seriously, I honor your technical expertise very much. But isn't it rather daily studio routine to capture each instrument as dry and isolated as possible* into a multitrack setup? That should be pretty far from what any of the participating artists has actually heard during the session (often a personal live mix over headphones).

* without necessarily having to record in separated spaces or takes.

Why Live-vs-Recorded Listening Tests Don't Work

Reply #12
It's unfortunate that these anecdotal experiences are completely subjective and were likely tainted by things that had nothing to do with actual sound.

Nothing unfortunate about having transcendent experiences involving sound. Eventually you've got to come down and be honest that, as there should be, there's much more to making these experiences than the sound.

Why Live-vs-Recorded Listening Tests Don't Work

Reply #13
Quite unfortunate if someone wants to garner something meaningful and objective in order to advance the topic at hand.  We have TOS #8 for a reason.

Why Live-vs-Recorded Listening Tests Don't Work

Reply #14
I also have trouble wrapping my head around the idea that those other things have "tainted" the experience.
...
But one can still to a large degree seperate the sound form the overall event. The cool factor certainly plays big time at a rock concert but it doesn't "fool" me into thinking that it sounds good.
...
etc.


<edit: shortened>
When seeing an expensive amp in action can already cause a huge difference compared to blind testing (many have been there), what do you expect from peer emotions, a light show, and star appeal?
</edit>

PS

Over several threads I have got the impression that you neglect anything, that could only remotely interfere with your own judgment about your own subjective experiences. You try to pull every discussion down to a level of particularity and dismiss any approaches to gain objective knowledge by means of abstraction or protocol. At best any experience as you perceive it, have perceived, or are going to perceive, is left untouched by anything that isn't rooted in your own holistic experience.

But people here are communicating with entirely different goals. They try to find out what aspects of subjective experience can be disregarded (or controlled by protocol or setup) for the proliferation of common, objective knowledge. Of course that can't be as rich and holistic as a singular, subjective experience. But that's not the goal, either. So continued bragging about the exclusivity of ones experience won't find much understanding here.

Why Live-vs-Recorded Listening Tests Don't Work

Reply #15
Unless he can assure us that his opinion about the relative sound quality of a live event cannot possibly be influenced by other factors than just sound quality then he hasn't a leg to stand on.

In case you aren't catching on, Scott, lofty esoteric comments about sound which are not backed by blind testing is generally frowned upon.

Why Live-vs-Recorded Listening Tests Don't Work

Reply #16
analogscott's post binned per TOS #2.  Further discussion that is not directly on topic with the first post resulting in off-topic disagreement will be binned and warnings will be issued.

Why Live-vs-Recorded Listening Tests Don't Work

Reply #17
While going to a concert in a good hall can be an exceptional experience, I have experienced the most detail in classical music from great records. If you want every detail, IMHO, the best records have surpassed live performances (which doesn't necessarily make it the better experience) for quite some time now. But experience is hard to quantify anyway, so why don't enjoy the best of both worlds?



This pretty much sums up my experience and position, too. Complete agreement, in fact.

Why Live-vs-Recorded Listening Tests Don't Work

Reply #18
But people here are communicating with entirely different goals. They try to find out what aspects of subjective experience can be disregarded (or controlled by protocol or setup) for the proliferation of common, objective knowledge. Of course that can't be as rich and holistic as a singular, subjective experience. But that's not the goal, either. So continued bragging about the exclusivity of ones experience won't find much understanding here.

And that nicely sums up the point of HA, as well as the argument in favour of the existence of this whole approach.

Why Live-vs-Recorded Listening Tests Don't Work

Reply #19
I also have trouble wrapping my head around the idea that those other things have "tainted" the experience.
...
But one can still to a large degree seperate the sound form the overall event. The cool factor certainly plays big time at a rock concert but it doesn't "fool" me into thinking that it sounds good.
...
etc.


<edit: shortened>
When seeing an expensive amp in action can already cause a huge difference compared to blind testing (many have been there), what do you expect from peer emotions, a light show, and star appeal?
</edit>

PS

Over several threads I have got the impression that you neglect anything, that could only remotely interfere with your own judgment about your own subjective experiences. You try to pull every discussion down to a level of particularity and dismiss any approaches to gain objective knowledge by means of abstraction or protocol. At best any experience as you perceive it, have perceived, or are going to perceive, is left untouched by anything that isn't rooted in your own holistic experience.

But people here are communicating with entirely different goals. They try to find out what aspects of subjective experience can be disregarded (or controlled by protocol or setup) for the proliferation of common, objective knowledge. Of course that can't be as rich and holistic as a singular, subjective experience. But that's not the goal, either. So continued bragging about the exclusivity of ones experience won't find much understanding here.



This thread deals with the topic of live music as a reference for judging speakers. In any such discussion we have to deal with a few axioms. 1. Live music is a reference for judging audio recording and playback 2. Accuracy in audio is the goal. My point was to examine the roots of these axioms and consider if they are valid and why. Sean raised a very intereting point when he brought up studio recordings that are productions unto themselves with no live acoustic performance to use as any sort of reference for live v. playback. He asked the following question "What about recordings that are not intended to sound like a live performance? That would include about 95% of all recording made today. How do we judge the accuracy of those?" To me this really segways into questioning the axiom that accuracy should be the goal in audio as well as questioning the validity of the axiom that live music should be the reference.

My discussion of my personal experience with live music was intended to address why anyone would want to use live music as a reference. I was really surprised to see the intense anger and hostility it stirred up not to mention the bizzarre interpretations of my intent. "Bragging about the excusivity of my experience?" No...not what I was doing. I sure didn't see my experiences with live music as a bragging point nor as anything "exclusive." Quite the opposite. I really thought others on this thread would relate to those sorts of experiences and would have had similar experiences. Guess not. so let's move on from there shall we? Lets just change all that to an assertion that personal experience with live music is a likely common motivating factor among those who wish to use live music as a reference for judging audio. OK?

Folks, I was calling into question these two basic axioms. They do underpin the question addressed in the OP. I would think any forum looking for "common objective knowledge" would welcome an examination of axioms. I really had no idea it would be so upsetting.  I wasn't trying to brag about anything.   

P.S. your impressions of me are *your impressions.* I won't go any further on that subject as we have all been warned to stay on topic and your opinion of *me* is off topic.

Why Live-vs-Recorded Listening Tests Don't Work

Reply #20
BTW, if you wish to speak about how great an artist so-and-so is, we have a forum for it.  Listening Tests is most certainly not that forum.

Why Live-vs-Recorded Listening Tests Don't Work

Reply #21
The reference is what the artist heard in the studio plain and simple. That is the "live performance"


Seriously, I honor your technical expertise very much. But isn't it rather daily studio routine to capture each instrument as dry and isolated as possible* into a multitrack setup? That should be pretty far from what any of the participating artists has actually heard during the session (often a personal live mix over headphones).

* without necessarily having to record in separated spaces or takes.


Yes, you are right, that is generally how most recordings are made today.

Sometimes the studio performance is captured live (e.g. jazz recordings) but that is less common.

It's the final mix with all the reverb, auto-tuning, EQ, 20:1 compression that constitutes the "reference" as heard by the main artist/producer. That would be the equivalent of the "live performance" that Edison and AR were trying to reproduce.

My point is that the "live performance" doesn't exist in most recordings today, and moreover accuracy is not necessarily the goal. But we still need some way reliably delivering that final mix to consumers so they hear what the artist intended.


Cheers
Sean Olive
Audio Musings

Why Live-vs-Recorded Listening Tests Don't Work

Reply #22
I have a couple of questions.

Quote
Live and Recorded Performances Must Be Identical

For live-versus-recorded tests to be valid, the live and recorded performance should be identical, having the same notes, intonation, tempo, dynamics, loudness, balance between instruments, and the same location and sense of space of the instruments. Otherwise, there are extraneous cues that allow listeners to readily identify the live and recorded performances. Midi-controlled instruments (e.g. player pianos) are but one example of how this problem could be resolved.


Would it be possible to design a valid test with the opposite approach? That is, instead of trying to reproduce a single performance identically, could we use various different performances and recordings every time?

I'm thinking of such scenario: suppose we need a test with 20 trials. Take 20 different singers with different voices, make 20 recordings. And then let some 20 more singers (again, all different voices) perform during the test, a sort of A/B test.
This way, I'm thinking, the singers don't even have to perform the same piece of music. It could be different music material every trial/performance.

Would it be possible to gather any statistically significant result from such a test?



And my 2nd question: can we consider our everyday practice of enjoying recorded music as the "ultimate" proof that such recordings are indeed capable of creating an illusion of live performance?
After several decades of such practical experience all across the globe, perhaps we already have enough evidence to draw some statistically valid conclusions? or still not?
I mean, okay, measuring the accuracy of a particular loudspeaker is one thing, but can't we say anything definitive of the technology in general?


1) I think a basic tenet of a good scientific experiment is that it is repeatable. So using humans musicians as sound sources is going to cause a lot of errors, and biases if the live performance doesn't perfectly match the recorded one. If you can devise a way to compare the live performance (via live mic feeds w. no delay) and compare that double-blind  to the performance  that would eliminate some of the errors.

I don't see how your method gets around this problem.  You can have 20 singers but unless their performances perfectly match their recordings then listeners have extraneous cues besides sound quality that tell them something is different.

2) I think it has been proven that most people can  pretty well enjoy music listening to any old piece of crap. I first became really aware of how sound  quality affects my enjoyment of music when someone recorded my piano recital with 2 mics located underneath the piano, and charged me for the tape. It didn't sound at all like how I played the piano  (really boomy and dull).  I was really pissed off. At that point I realized the importance of sound recording and reproduction and decided to pursue it as a career. But most people probably never think about it, until they go to a really bad live concert, and the artist doesn't sound anything like the recordings. Even then, quantity usually matters more than quality (think rock n'roll).

Cheers
Sean Olive
Audio Musings

Why Live-vs-Recorded Listening Tests Don't Work

Reply #23
1) I think a basic tenet of a good scientific experiment is that it is repeatable. So using humans musicians as sound sources is going to cause a lot of errors, and biases if the live performance doesn't perfectly match the recorded one. If you can devise a way to compare the live performance (via live mic feeds w. no delay) and compare that double-blind  to the performance  that would eliminate some of the errors.

I don't see how your method gets around this problem.


It turned out to be the RCT methods that I was thinking of. Please see above, in my second post in this thread (click here).

The way I see it, RCT does not require identical stimuli, so this whole problem becomes a non-issue in RCT. No need for any "live mic feeds" or anything exotic.
Am I wrong? Why?


However, I must emphasize one important point:

When we think of 'live-vs-recorded' tests it is quite natural to require that the recorded sound must be indistinguishable from the corresponding live sound. That is what we would want to test,  naturally. And that is, I suppose, what you are aiming at in the article. And we all know that ABX sort of tests are the best to get it done. And then, yes, we would run into all sorts of problems that would make such 'live-vs-recorded' tests practically impossible, especially w.r.t. the need for identical repeatable stimuli.

But then, if we still want to make any sort of  'live-vs-recorded' comparison, what is the next best thing we could do?

We could relax the requirements. Instead of demanding indistinguishable sound, we could ask ourselves: is our audio system able to create an illusion of a live performance?

Please note that it is a more broad, less strict demand.
Instead of demanding the system to identically reproduce some particular performance on some particular occasion, here we ask if the reproduction could be mistaken for a live performance, or not at all. I think this question can be answered by using 'explanatory' RCT methods.

Is this a meaningless question to test? I think not. I think it is a meaningful question, especially from the consumer perspective.
With that in mind, I think that useful 'live-vs-recorded' tests might be possible - by using RCT methods.


(think rock n'roll).

I think it would be wise, in the context of this discussion, to limit our definition of "live performance" to acoustic instruments and human voice.
Using mic feeds/PA etc would effectively degenerate a 'live-vs-recorded' test into a 'recorded-vs-recorded'. We don't want that.


2) I think it has been proven that most people can  pretty well enjoy music listening to any old piece of crap.


We should ask them not how they enjoy it, but whether the sound is realistic enough to be mistaken for a live performance. There are plenty of classical musicians who listen to all sorts of classical records through quite decent equipment. We could ask them.
Oh, and by the way, in the RCT vocabulary, this would be the 'pragmatic' type of RCT, 'in-the-field' trials.
Would you dismiss any 'pragmatic' type of RCT as unscientific?

Why Live-vs-Recorded Listening Tests Don't Work

Reply #24
Would it be possible to design a valid test with the opposite approach? That is, instead of trying to reproduce a single performance identically, could we use various different performances and recordings every time?


A quick follow-up on my first question.

I did some googling and found that I was actually thinking of a kind of test called "Randomized controlled trial" (RCT).
The "explanatory" type of RCT with "parallel-group" design and "allocation concealment", in particular. The goal of such RCT is to test the 'efficacy' of a treatment or medicine given to a group of patients.

So, here goes my analogy:
* 'efficacy' = ability to create illusion of a live performance.
* Participants (patients) are the singers/performers.
* Half of the participants are allocated to receive the 'treatment' (record and playback via loudspeakers),
and the other half is allocated to receive no 'treatment' (perform live).

The problem, I guess, is that it might be not quite properly triple-blind, since our 'participants' (singers) know which 'treatment' they are receiving, obviously.

But maybe this bias could be eliminated, too: let's make all singers perform live, but let some of them ("control group") perform before dummy-listeners in one room, and the other group perform before the actual audience in another room. Or something like that. Mix them and confuse them.
Then the singers wouldn't know which of their performances would actually count, and the test becomes fully triple-blind, I hope. 

So, on the surface it seems that it should be possible to eliminate the need for identical stimuli in a live-vs-recorded test - by using RCT method.

Any thoughts? Anybody? 


The participants in a listening test are normally the listeners - not the singers/performers, who act as one of the stimuli.

It is the listeners who decide whether or not the reproduction of the recording is similar to the live performance -- not the performers. I think you have misunderstood the original intent of the live-vs-recorded test.