IPB

Welcome Guest ( Log In | Register )

Why Live-vs-Recorded Listening Tests Don't Work
solive
post Jul 10 2010, 11:08
Post #1





Group: Members
Posts: 162
Joined: 21-February 04
From: Los Angeles
Member No.: 12173



Thomas Edison was probably the greatest stereo salesman that ever lived. He believed that "listeners will hear what you tell them to hear", and he was pretty successful convincing thousands of listeners that his 1910 Edison Diamond Disc Phonograph reproduced recordings that sounded identical to a live performance. His secret weapon was an elaborate live-versus-recorded demonstration that managed to convince people that his phonograph sounded a lot better than it really was.

Several times over the past 10 years, I have been asked by live-versus-recorded apologists why I don't do these types of the tests since they claim they are the only true valid measures of loudspeaker fidelity or accuracy. That is what prompted me to write about why I believe live-versus-recorded listening tests don't work, in this month's blog article

Cheers
Sean Olive
Audio Musings

This post has been edited by solive: Jul 10 2010, 11:30


--------------------
Sean Olive
[url="http://seanolive.com"]Audio Musings[/url]
Go to the top of the page
+Quote Post
 
Start new topic
Replies
kdo
post Jul 10 2010, 16:53
Post #2





Group: Members (Donating)
Posts: 304
Joined: 18-April 02
From: Russia
Member No.: 1812



I have a couple of questions.

QUOTE
Live and Recorded Performances Must Be Identical

For live-versus-recorded tests to be valid, the live and recorded performance should be identical, having the same notes, intonation, tempo, dynamics, loudness, balance between instruments, and the same location and sense of space of the instruments. Otherwise, there are extraneous cues that allow listeners to readily identify the live and recorded performances. Midi-controlled instruments (e.g. player pianos) are but one example of how this problem could be resolved.


Would it be possible to design a valid test with the opposite approach? That is, instead of trying to reproduce a single performance identically, could we use various different performances and recordings every time?

I'm thinking of such scenario: suppose we need a test with 20 trials. Take 20 different singers with different voices, make 20 recordings. And then let some 20 more singers (again, all different voices) perform during the test, a sort of A/B test.
This way, I'm thinking, the singers don't even have to perform the same piece of music. It could be different music material every trial/performance.

Would it be possible to gather any statistically significant result from such a test?



And my 2nd question: can we consider our everyday practice of enjoying recorded music as the "ultimate" proof that such recordings are indeed capable of creating an illusion of live performance?
After several decades of such practical experience all across the globe, perhaps we already have enough evidence to draw some statistically valid conclusions? or still not?
I mean, okay, measuring the accuracy of a particular loudspeaker is one thing, but can't we say anything definitive of the technology in general?

This post has been edited by kdo: Jul 10 2010, 16:55
Go to the top of the page
+Quote Post
solive
post Jul 12 2010, 06:46
Post #3





Group: Members
Posts: 162
Joined: 21-February 04
From: Los Angeles
Member No.: 12173



QUOTE (kdo @ Jul 10 2010, 08:53) *
I have a couple of questions.

QUOTE
Live and Recorded Performances Must Be Identical

For live-versus-recorded tests to be valid, the live and recorded performance should be identical, having the same notes, intonation, tempo, dynamics, loudness, balance between instruments, and the same location and sense of space of the instruments. Otherwise, there are extraneous cues that allow listeners to readily identify the live and recorded performances. Midi-controlled instruments (e.g. player pianos) are but one example of how this problem could be resolved.


Would it be possible to design a valid test with the opposite approach? That is, instead of trying to reproduce a single performance identically, could we use various different performances and recordings every time?

I'm thinking of such scenario: suppose we need a test with 20 trials. Take 20 different singers with different voices, make 20 recordings. And then let some 20 more singers (again, all different voices) perform during the test, a sort of A/B test.
This way, I'm thinking, the singers don't even have to perform the same piece of music. It could be different music material every trial/performance.

Would it be possible to gather any statistically significant result from such a test?



And my 2nd question: can we consider our everyday practice of enjoying recorded music as the "ultimate" proof that such recordings are indeed capable of creating an illusion of live performance?
After several decades of such practical experience all across the globe, perhaps we already have enough evidence to draw some statistically valid conclusions? or still not?
I mean, okay, measuring the accuracy of a particular loudspeaker is one thing, but can't we say anything definitive of the technology in general?


1) I think a basic tenet of a good scientific experiment is that it is repeatable. So using humans musicians as sound sources is going to cause a lot of errors, and biases if the live performance doesn't perfectly match the recorded one. If you can devise a way to compare the live performance (via live mic feeds w. no delay) and compare that double-blind to the performance that would eliminate some of the errors.

I don't see how your method gets around this problem. You can have 20 singers but unless their performances perfectly match their recordings then listeners have extraneous cues besides sound quality that tell them something is different.

2) I think it has been proven that most people can pretty well enjoy music listening to any old piece of crap. I first became really aware of how sound quality affects my enjoyment of music when someone recorded my piano recital with 2 mics located underneath the piano, and charged me for the tape. It didn't sound at all like how I played the piano (really boomy and dull). I was really pissed off. At that point I realized the importance of sound recording and reproduction and decided to pursue it as a career. But most people probably never think about it, until they go to a really bad live concert, and the artist doesn't sound anything like the recordings. Even then, quantity usually matters more than quality (think rock n'roll).

Cheers
Sean Olive
Audio Musings

This post has been edited by solive: Jul 12 2010, 06:47


--------------------
Sean Olive
[url="http://seanolive.com"]Audio Musings[/url]
Go to the top of the page
+Quote Post
Arnold B. Kruege...
post Jul 13 2010, 17:25
Post #4





Group: Members
Posts: 4474
Joined: 29-October 08
From: USA, 48236
Member No.: 61311



QUOTE (solive @ Jul 12 2010, 01:46) *
1) I think a basic tenet of a good scientific experiment is that it is repeatable. So using humans musicians as sound sources is going to cause a lot of errors, and biases if the live performance doesn't perfectly match the recorded one. If you can devise a way to compare the live performance (via live mic feeds w. no delay) and compare that double-blind to the performance that would eliminate some of the errors.


The non-repeatability of live performances was illustrated to me by the following experience:

Some years back some studio techs prepared and sold sets of CDs that were designed to illustrate the characteristic colorations of microphones and mic preamps. I invensted in a set.

I decided to see what would happen if I tried to ABX them. In the process of preparing the samples for ABXing, I found that hte purportedly identical musical samples that were supposed to differ only in terms of equipment used were different in fairly gross ways. The musical samples had different lengths if you trimmed them to be musically alike. Their average levels varied by more than enough to be audible. Once those basic issues were dealt with, there were still clearly audible differences in timing, inflection and intonation that were clearly audible. I never had any trouble ABXing them and obtaining perfect or nearly perfect scores in short order based on just the misical differences.

The second issue is that the musical reproduction chain can be broken down into three general areas being microphones and microphone technique, audio signal stoarge and production, and speakers and room acoustics. By various means we can show that signal storage and production can be sonically transparent. It is well known that neither of the other two areas of music reproduction have attained that level of refinement.
Go to the top of the page
+Quote Post
zane9
post Jul 13 2010, 18:50
Post #5





Group: Members
Posts: 13
Joined: 19-May 09
Member No.: 69939



QUOTE (Arnold B. Krueger @ Jul 13 2010, 11:25) *
...The second issue is that the musical reproduction chain can be broken down into three general areas being microphones and microphone technique, audio signal stoarge and production, and speakers and room acoustics. By various means we can show that signal storage and production can be sonically transparent. It is well known that neither of the other two areas of music reproduction have attained that level of refinement.


Not withstanding the deficiences of the other two areas mentioned by Arnold, my default preference is listening to a recording of unamplified music in a non-studio setting, done with a pair of microphones and no mix.

Not so easy to find these recordings, these days.

Go to the top of the page
+Quote Post
Arnold B. Kruege...
post Jul 13 2010, 20:21
Post #6





Group: Members
Posts: 4474
Joined: 29-October 08
From: USA, 48236
Member No.: 61311



QUOTE (zane9 @ Jul 13 2010, 13:50) *
QUOTE (Arnold B. Krueger @ Jul 13 2010, 11:25) *
...The second issue is that the musical reproduction chain can be broken down into three general areas being microphones and microphone technique, audio signal stoarge and production, and speakers and room acoustics. By various means we can show that signal storage and production can be sonically transparent. It is well known that neither of the other two areas of music reproduction have attained that level of refinement.


Not withstanding the deficiences of the other two areas mentioned by Arnold, my default preference is listening to a recording of unamplified music in a non-studio setting, done with a pair of microphones and no mix.


I've made in excess of 500 recordings of live, unamplified music using 2 microphones chosen for flat on-axis response, no equalization or other processsing, for hire, in just the past 5 years.

Changing just the position and orientation of the 2 microphones, I can adjust the timbre and soundstanging of the recording over a fairly wide range. My choices are informed by the desires of the clients, who are professional musicans, mostly high school and college educators. I'm usually tryng to duplicqte the sound in a particular range of locations in the auditorium.

In no case would I consider the resulting recordings to be "sonically accurate" in the sense that they would frustrate or even challenge attempts at identification in an ABX test. I don't think they would be very hard to differentiate from live sound in a live versus recorded comparison.

I've also made a goodly number of multitrack recordings using close micing, distant micing and even the capture of raw electical signals from amplified electronic instruments. I think that most people would find carefully mixed recordings made this way to *not* be obviously less "lifielike" than the ones made using minimal micing and no mixing or other processing.

In terms of recreation of lifelike sound, the prcedure that seems to get the closest is IME close-micing and loudspeaker reproduction in the same room, given that the room is extremely reverberant.
Go to the top of the page
+Quote Post
kdo
post Jul 14 2010, 02:19
Post #7





Group: Members (Donating)
Posts: 304
Joined: 18-April 02
From: Russia
Member No.: 1812



I sense a big fat TOS-8 violation right here:
QUOTE (Arnold B. Krueger @ Jul 13 2010, 21:21) *
In terms of recreation of lifelike sound, the prcedure that seems to get the closest is IME close-micing and loudspeaker reproduction in the same room, given that the room is extremely reverberant.

Can you back up that assertion by a rigorous 'live-vs-recorded' test with statistically significant results?


Hint: this is exactly why I believe that 'live-vs-recorded' tests are necessary. To validate any such claims about "lifelike sound".

Go to the top of the page
+Quote Post
Ed Seedhouse
post Jul 14 2010, 02:35
Post #8





Group: Members
Posts: 175
Joined: 19-May 09
Member No.: 69959



One form of "live vs. reccorded" test with less problems than others might be to have a high quality speaker recorded playing music in an anechoic chamber, then seeing how much another speaker would sound like it when they both play in the same room.

So we record speaker A playing musical recordings in an anechoic chamber. Then we listen in a room to speaker "B" playing the recording made in the chamber and comparing it with speaker A playing the original recordens. Assuming speaker A has some colorations, how well would speaker B do in playing these colorations accurately?

We could even use speaker A as it's own comparitor. If it could play it's own self back unchanged vs. the original recordings then we would know it was certainly highly accurate. In fact I doubt if any actual production speaker could pass that test, but it would be great to be proven wrong.

This would at least be much easier to arrange than with live musicians.


--------------------
Ed Seedhouse
Go to the top of the page
+Quote Post

Posts in this topic
- solive   Why Live-vs-Recorded Listening Tests Don't Work   Jul 10 2010, 11:08
- - kdo   I have a couple of questions. QUOTE Live and Reco...   Jul 10 2010, 16:53
|- - kdo   QUOTE (kdo @ Jul 10 2010, 17:53) Would it...   Jul 11 2010, 16:25
||- - solive   QUOTE (kdo @ Jul 11 2010, 08:25) QUOTE (k...   Jul 13 2010, 00:43
|||- - kdo   QUOTE (solive @ Jul 13 2010, 01:43) The p...   Jul 13 2010, 01:02
||||- - solive   QUOTE (kdo @ Jul 12 2010, 17:02) QUOTE (s...   Jul 13 2010, 04:09
|||- - Arnold B. Krueger   QUOTE (solive @ Jul 12 2010, 19:43) It is...   Jul 13 2010, 20:26
||- - solive   QUOTE (kdo @ Jul 11 2010, 08:25) QUOTE (k...   Jul 13 2010, 04:53
||- - kdo   QUOTE (solive @ Jul 13 2010, 05:09) Sorry...   Jul 13 2010, 20:10
|- - solive   QUOTE (kdo @ Jul 10 2010, 08:53) I have a...   Jul 12 2010, 06:46
|- - kdo   QUOTE (solive @ Jul 12 2010, 07:46) 1) I ...   Jul 12 2010, 21:25
|- - Arnold B. Krueger   QUOTE (solive @ Jul 12 2010, 01:46) 1) I ...   Jul 13 2010, 17:25
|- - zane9   QUOTE (Arnold B. Krueger @ Jul 13 2010, 11...   Jul 13 2010, 18:50
||- - Arnold B. Krueger   QUOTE (zane9 @ Jul 13 2010, 13:50) QUOTE ...   Jul 13 2010, 20:21
||- - kdo   I sense a big fat TOS-8 violation right here: QUOT...   Jul 14 2010, 02:19
||- - Ed Seedhouse   One form of "live vs. reccorded" test wi...   Jul 14 2010, 02:35
|||- - kdo   QUOTE (Ed Seedhouse @ Jul 14 2010, 03:35)...   Jul 14 2010, 02:57
||- - Arnold B. Krueger   QUOTE (kdo @ Jul 13 2010, 21:19) I sense ...   Jul 14 2010, 03:12
||- - kdo   QUOTE (Arnold B. Krueger @ Jul 14 2010, 04...   Jul 14 2010, 03:30
||- - Arnold B. Krueger   QUOTE (kdo @ Jul 13 2010, 22:30) QUOTE (A...   Jul 14 2010, 10:42
||- - kdo   QUOTE (Arnold B. Krueger @ Jul 14 2010, 11...   Jul 14 2010, 16:56
||- - Arnold B. Krueger   QUOTE (kdo @ Jul 14 2010, 11:56) QUOTE (A...   Jul 14 2010, 17:15
||- - kdo   QUOTE (Arnold B. Krueger @ Jul 14 2010, 18...   Jul 14 2010, 17:58
|- - 2Bdecided   QUOTE (Arnold B. Krueger @ Jul 13 2010, 17...   Jul 14 2010, 19:04
|- - analog scott   QUOTE (2Bdecided @ Jul 14 2010, 19:04) QU...   Jul 14 2010, 20:40
|- - Arnold B. Krueger   QUOTE (2Bdecided @ Jul 14 2010, 14:04) It...   Jul 16 2010, 07:58
|- - 2Bdecided   QUOTE (Arnold B. Krueger @ Jul 16 2010, 07...   Jul 16 2010, 16:29
|- - Arnold B. Krueger   QUOTE (2Bdecided @ Jul 16 2010, 11:29) Cr...   Jul 16 2010, 23:59
- - analog scott   QUOTE (solive @ Jul 10 2010, 12:08) Thoma...   Jul 10 2010, 18:41
|- - solive   QUOTE (analog scott @ Jul 10 2010, 10:41)...   Jul 11 2010, 07:46
|- - analog scott   QUOTE (solive @ Jul 11 2010, 07:46) What ...   Jul 11 2010, 15:50
|- - googlebot   QUOTE (analog scott @ Jul 11 2010, 16:50)...   Jul 11 2010, 16:53
||- - analog scott   QUOTE (googlebot @ Jul 11 2010, 16:53) QU...   Jul 11 2010, 17:44
||- - aclo   QUOTE (googlebot @ Jul 11 2010, 17:53) Wh...   Jul 12 2010, 04:06
|- - solive   QUOTE (analog scott @ Jul 11 2010, 07:50)...   Jul 11 2010, 20:16
||- - analog scott   QUOTE (solive @ Jul 11 2010, 21:16) QUOTE...   Jul 11 2010, 20:33
||- - googlebot   QUOTE (solive @ Jul 11 2010, 21:16) The r...   Jul 11 2010, 20:33
|||- - solive   QUOTE (googlebot @ Jul 11 2010, 12:33) QU...   Jul 12 2010, 06:19
||- - 2Bdecided   QUOTE (solive @ Jul 11 2010, 20:16) The r...   Jul 14 2010, 18:54
|- - greynol   QUOTE (analog scott @ Jul 11 2010, 07:50)...   Jul 11 2010, 20:28
|- - Notat   QUOTE (greynol @ Jul 11 2010, 13:28) It...   Jul 11 2010, 23:06
- - greynol   Quite unfortunate if someone wants to garner somet...   Jul 11 2010, 23:19
- - greynol   Unless he can assure us that his opinion about the...   Jul 12 2010, 02:26
- - greynol   analogscott's post binned per TOS #2. Further...   Jul 12 2010, 03:09


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 22nd December 2014 - 20:37