IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
More Evidence that Kids (American and Japanese) Prefer Accurate Sound, Blind tests on CD/Mp3 and Loudspeaker Preference
solive
post May 10 2012, 22:36
Post #1





Group: Members
Posts: 162
Joined: 21-February 04
From: Los Angeles
Member No.: 12173



This might of interest to people who followed the media hype the last 3 years surrounding the notion that kids prefer low-quality MP3 versus lossless CD sound reproduction based on an unpublished study by Jonathan Berger that never made scrutinized by the scientific community.

This past month I presented at paper at the 132nd AES Convention in Budapest that summarized some follow-up research where I tested 58 high school and college kids to determine their preference choices for CD-quality versus MP3 (128 kbps) music selections. as well as their loudspeaker preferences. I also ran 149 native speaking Japanese students through the same loudspeaker test to see if cross-cultural differences possibly influence sound quality preferences.

The findings and slide presentation can be found in my blog and S&V has written a nice commentary.

This post has been edited by solive: May 10 2012, 22:38


--------------------
Sean Olive
[url="http://seanolive.com"]Audio Musings[/url]
Go to the top of the page
+Quote Post
hellokeith
post May 11 2012, 00:26
Post #2





Group: Members
Posts: 288
Joined: 14-August 06
Member No.: 34027



In the loudspeaker results, I don't see A/B/C/D labeled to which speaker. Can you provide that?
Go to the top of the page
+Quote Post
solive
post May 11 2012, 04:54
Post #3





Group: Members
Posts: 162
Joined: 21-February 04
From: Los Angeles
Member No.: 12173



QUOTE (hellokeith @ May 10 2012, 16:26) *
In the loudspeaker results, I don't see A/B/C/D labeled to which speaker. Can you provide that?


That was done purposedly to remove the identities of the loudspeakers from the test results. That is typical protocol when publishing papers in the AES.


--------------------
Sean Olive
[url="http://seanolive.com"]Audio Musings[/url]
Go to the top of the page
+Quote Post
mzil
post May 11 2012, 06:59
Post #4





Group: Members
Posts: 606
Joined: 5-August 07
Member No.: 45913



"More evidence that kids ..." ?

Out of curiosity, what's the "pre existing" evidence which makes your study the "more"?

Thanks, and good to have you in the forum, BTW.

This post has been edited by mzil: May 11 2012, 07:58
Go to the top of the page
+Quote Post
2Bdecided
post May 11 2012, 10:10
Post #5


ReplayGain developer


Group: Developer
Posts: 5139
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



A good study and a great message to the industry.

If only you sold Infinity speakers in the UK! sad.gif

Cheers,
David.
Go to the top of the page
+Quote Post
Arnold B. Kruege...
post May 11 2012, 11:12
Post #6





Group: Members
Posts: 3797
Joined: 29-October 08
From: USA, 48236
Member No.: 61311



QUOTE (hellokeith @ May 10 2012, 19:26) *
In the loudspeaker results, I don't see A/B/C/D labeled to which speaker. Can you provide that?


Link to an independent technical analysis of the .pdf suggesting loudspeaker identity information

"Looking at the measurements from the .pdf you note that the speaker in question is a 3-way (note the local directivity maxima around 350Hz and 3KHz) with a small midrange.

(From the Infinity site page for the P362: Crossover Frequency(ies) 350Hz, 3,300Hz;)

The last speaker is the Martin-Logan. Directivity is constant due to dipole cancellation starting at 300Hz until the panel becomes acoustically large. You can see the dipole nulls on-axis, panel resonances that show up as similar bumps in all the curves, and some lobing.

The third one should be the Klipsch with the wave guide providing fairly constant directivity from 1Khz on up.

The second should be the Polk where it looks like there's a cross-over at 200-300Hz and a second around 2-3KHz. The large midrange beams more than the smaller unit on the Infinity.
"

This post has been edited by Arnold B. Krueger: May 11 2012, 11:13
Go to the top of the page
+Quote Post
Arnold B. Kruege...
post May 11 2012, 11:16
Post #7





Group: Members
Posts: 3797
Joined: 29-October 08
From: USA, 48236
Member No.: 61311



QUOTE (2Bdecided @ May 11 2012, 05:10) *
A good study and a great message to the industry.

If only you sold Infinity speakers in the UK! sad.gif


They don't?

Harman Consumer UK St. Albans, Hertfordshire +44 1707 278100 +44 1707 278129 info.uk@harman.com

Amazon UK seems to have Infinity products, I don't know how much.
Go to the top of the page
+Quote Post
Arnold B. Kruege...
post May 11 2012, 11:23
Post #8





Group: Members
Posts: 3797
Joined: 29-October 08
From: USA, 48236
Member No.: 61311



QUOTE (solive @ May 10 2012, 23:54) *
QUOTE (hellokeith @ May 10 2012, 16:26) *
In the loudspeaker results, I don't see A/B/C/D labeled to which speaker. Can you provide that?


That was done purposedly to remove the identities of the loudspeakers from the test results. That is typical protocol when publishing papers in the AES.


Is there another online document that contains the on-axis frequency response curve of a P362?
Go to the top of the page
+Quote Post
2Bdecided
post May 11 2012, 13:17
Post #9


ReplayGain developer


Group: Developer
Posts: 5139
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



QUOTE (Arnold B. Krueger @ May 11 2012, 11:16) *
QUOTE (2Bdecided @ May 11 2012, 05:10) *
A good study and a great message to the industry.

If only you sold Infinity speakers in the UK! sad.gif


They don't?

Harman Consumer UK St. Albans, Hertfordshire +44 1707 278100 +44 1707 278129 info.uk@harman.com

Amazon UK seems to have Infinity products, I don't know how much.
On Amazon, there's a third party marketplace seller, with a single pair of old stock speakers available at several times the US market rate.

Harmon Kardon is well known and stocked in the UK. The Infinity brand is almost non existent.

I'd already guessed the identity of speaker A wink.gif


One way of reading Sean's results is that only 20% of listeners could hear anything wrong with 128kbps mp3. To be fair, I wouldn't stake my life on being able to hear anything wrong with 128kbps mp3 either.

Cheers,
David.
Go to the top of the page
+Quote Post
solive
post May 11 2012, 14:46
Post #10





Group: Members
Posts: 162
Joined: 21-February 04
From: Los Angeles
Member No.: 12173



QUOTE (2Bdecided @ May 11 2012, 05:17) *
QUOTE (Arnold B. Krueger @ May 11 2012, 11:16) *
QUOTE (2Bdecided @ May 11 2012, 05:10) *
A good study and a great message to the industry.

If only you sold Infinity speakers in the UK! sad.gif


They don't?

Harman Consumer UK St. Albans, Hertfordshire +44 1707 278100 +44 1707 278129 info.uk@harman.com

Amazon UK seems to have Infinity products, I don't know how much.
On Amazon, there's a third party marketplace seller, with a single pair of old stock speakers available at several times the US market rate.

Harmon Kardon is well known and stocked in the UK. The Infinity brand is almost non existent.

I'd already guessed the identity of speaker A wink.gif


One way of reading Sean's results is that only 20% of listeners could hear anything wrong with 128kbps mp3. To be fair, I wouldn't stake my life on being able to hear anything wrong with 128kbps mp3 either.

Cheers,
David.


This was a preference test where listeners had to decide which one they preferred -- not an ABX test where the aim is to determine whether they can detect any audible difference. So just because some listeners could not reliably (i.e. 75% or 9/12 of the trials) indicate a preference for one of the choices doesn't imply they couldn't reliably detect a difference or "hear anything wrong with the 128 kbps MP3".




--------------------
Sean Olive
[url="http://seanolive.com"]Audio Musings[/url]
Go to the top of the page
+Quote Post
Arnold B. Kruege...
post May 11 2012, 16:08
Post #11





Group: Members
Posts: 3797
Joined: 29-October 08
From: USA, 48236
Member No.: 61311



QUOTE (2Bdecided @ May 11 2012, 08:17) *
One way of reading Sean's results is that only 20% of listeners could hear anything wrong with 128kbps mp3. To be fair, I wouldn't stake my life on being able to hear anything wrong with 128kbps mp3 either.


Another way to read it is that 40% couldn't hear a difference and half picked CD and half picked MP3. Many other numerical permutations involving even higher rates of guessing are possible, but this one came to mind first.
Go to the top of the page
+Quote Post
2Bdecided
post May 11 2012, 18:58
Post #12


ReplayGain developer


Group: Developer
Posts: 5139
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



That's an interesting question - the stats of this.

In a forced choice test where something is definitively worse, you know guessing which is worse (i.e. can't hear a difference) will give 50/50 (eventually), and anything beyond that shows the real effect.

When people have the option to say "no preference", but can state a preference without any proof that they really detect any difference, yet are being asked a question where the actual preference if they do hear a difference could go either way, I don't know what the stats are to weed out the guessers.

I can give the results from one real test I've been involved in though. 539 respondents. Two clips. Question was "which is better, A or B". In one task, both clips were in fact identical. 20% said "clip A is better", 58% said there is no difference, and 22% said "clip B is better".

In another test where clip A had its high frequencies trashed, 19% said clip A is better, 39% said there is no difference, and 42% said Clip B is better.

I always took that result to mean that about 20% could detect the damage done to the high frequencies, but I guess there are other interpretations even here.


Whatever, the first test proved a useful control sample IMO. Whether you need it or not depends on what results you get, and what conclusion you want to draw from them.

(I'm not criticising the current test - I'm agreeing with Sean + Arny that you can't definitively draw the conclusion that I tried to draw!)

Cheers,
David.
Go to the top of the page
+Quote Post
solive
post May 11 2012, 19:15
Post #13





Group: Members
Posts: 162
Joined: 21-February 04
From: Los Angeles
Member No.: 12173



QUOTE (Arnold B. Krueger @ May 11 2012, 08:08) *
QUOTE (2Bdecided @ May 11 2012, 08:17) *
One way of reading Sean's results is that only 20% of listeners could hear anything wrong with 128kbps mp3. To be fair, I wouldn't stake my life on being able to hear anything wrong with 128kbps mp3 either.


Another way to read it is that 40% couldn't hear a difference and half picked CD and half picked MP3. Many other numerical permutations involving even higher rates of guessing are possible, but this one came to mind first.


No, that's not quite accurate.

43% of the 58 students had a significant preference for CD meaning they chose it in 75% of the trials or greater (p <= 0.05)
52% of the students preferred CD in 50% to <75% of the trials.
Only 5% of the students made preference choices for CD in less than 50% of the trials.
0% of the students had a significant preference for MP3 meaning they preferred MP3 in >75% of the trials. This is in stark contrast to Berger's findings where he claims many students preferred MP3 @ 128 kbp/s to CD.

This post has been edited by solive: May 11 2012, 19:20


--------------------
Sean Olive
[url="http://seanolive.com"]Audio Musings[/url]
Go to the top of the page
+Quote Post
solive
post May 11 2012, 23:31
Post #14





Group: Members
Posts: 162
Joined: 21-February 04
From: Los Angeles
Member No.: 12173



QUOTE (2Bdecided @ May 11 2012, 10:58) *
That's an interesting question - the stats of this.

In a forced choice test where something is definitively worse, you know guessing which is worse (i.e. can't hear a difference) will give 50/50 (eventually), and anything beyond that shows the real effect.

When people have the option to say "no preference", but can state a preference without any proof that they really detect any difference, yet are being asked a question where the actual preference if they do hear a difference could go either way, I don't know what the stats are to weed out the guessers.

I can give the results from one real test I've been involved in though. 539 respondents. Two clips. Question was "which is better, A or B". In one task, both clips were in fact identical. 20% said "clip A is better", 58% said there is no difference, and 22% said "clip B is better".

In another test where clip A had its high frequencies trashed, 19% said clip A is better, 39% said there is no difference, and 42% said Clip B is better.

I always took that result to mean that about 20% could detect the damage done to the high frequencies, but I guess there are other interpretations even here.


Whatever, the first test proved a useful control sample IMO. Whether you need it or not depends on what results you get, and what conclusion you want to draw from them.

(I'm not criticising the current test - I'm agreeing with Sean + Arny that you can't definitively draw the conclusion that I tried to draw!)

Cheers,
David.


Agreed. This was a forced choice preference task that assumed people could hear differences. I tried to pick programs where the artifacts were quite obvious but some listeners are not very attentive. Some students told me they preferred the MP3 for applause because it rolled off the highs which they felt sounded too aggressive in the CD version. The audible differences between MP3 and CD with these tracks was not so subtle.

But there is nothing in the test design to weed out guessers. You can make the assumption, as you point out, that if they are guessing which one they prefer, the preference choices would eventually be equally distributed between CD and MP3 50-50%. But another possibility is that there is a program interaction and they prefer MP3 with 1/2 of the programs and CD with the other 1/2. There was no indication of this interaction in the results.



If I were to redo these tests I would probably use a hybrid ABX where they state the preference choice for A vs. B and indicate what X is. That would weed out the guessers and still give you some preference data. I've done this sort of test when comparing amplifiers, which sounded much more similar than MP3 vs CD.

You live and learn.

This post has been edited by solive: May 11 2012, 23:37


--------------------
Sean Olive
[url="http://seanolive.com"]Audio Musings[/url]
Go to the top of the page
+Quote Post
solive
post May 11 2012, 23:45
Post #15





Group: Members
Posts: 162
Joined: 21-February 04
From: Los Angeles
Member No.: 12173



QUOTE (2Bdecided @ May 11 2012, 10:58) *
That's an interesting question - the stats of this.


In another test where clip A had its high frequencies trashed, 19% said clip A is better, 39% said there is no difference, and 42% said Clip B is better.

I always took that result to mean that about 20% could detect the damage done to the high frequencies, but I guess there are other interpretations even here.


Whatever, the first test proved a useful control sample IMO. Whether you need it or not depends on what results you get, and what conclusion you want to draw from them.

(I'm not criticising the current test - I'm agreeing with Sean + Arny that you can't definitively draw the conclusion that I tried to draw!)

Cheers,
David.


If 42% said Clip B is better I would interpret that as meaning 42% (not 20%) can detect the damage done to Clip A. No? Is that what you meant to write?


--------------------
Sean Olive
[url="http://seanolive.com"]Audio Musings[/url]
Go to the top of the page
+Quote Post
benski
post May 12 2012, 02:10
Post #16


Winamp Developer


Group: Developer
Posts: 670
Joined: 17-July 05
From: Brooklyn, NY
Member No.: 23375



There were some minor problems with the test. These we discussed after the presentation, so you are already aware of these, Sean.

1) Except for the expert listeners in the speaker test, all these tests were "group tests". This means that each individual listener couldn't switch between A/B at their choice. It's likely that doing individual listening tests would have revealed stronger differentiation between CD/MP3 and amongst the different speakers.
2) For the MP3 listening test, listeners were given information about what sorts of artifacts to expect. This could have biased the listeners towards preferring CD over MP3.

Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 21st September 2014 - 13:48