IPB

Welcome Guest ( Log In | Register )

High Bitrate ~320kbps MP3 vs. Lossless audio, Spoiler: mp3s were TRANSCODED from Lame free-format.
Archimago
post Jan 15 2013, 21:11
Post #1





Group: Members
Posts: 13
Joined: 22-December 12
Member No.: 105342



Happy New Year and greetings everyone! I know the folks here at Hydrogen audio have been around and back with regards to these tests... Nonetheless, I've been collecting data from around the audio forums since early Dec 2012.

Since early December 2012, I've opened up a survey to see if music lovers & "audiophiles" around the world can tell the difference between high bitrate ~320kbps MP3 against the original un-lossy-compressed CD audio. I'm only planning to collect data until the end of January 2013.

So far, I have >70 responses from around the world with 5 continents represented. Folks have been using everything from inexpensive (but good) headphones to megabuck separates >$50K for this. There's 2 weeks left so if you've ever been curious about participating in a blind test or just want to add to the dataset, here's your chance :-).

The "test" is relatively simple and consists of 3 musical passages encoded as "Set A" vs. the same songs "Set B"; one of which was MP3 encoded.

Come to my blog to download the music and fill out the survey:
Archimago's Musings

Thanks and have a wonderful 2013!

OOps - the title should have read 'ends Jan 31'... [fixed]

This post has been edited by db1989: Jan 15 2013, 21:16
Go to the top of the page
+Quote Post
 
Start new topic
Replies
Dynamic
post Feb 4 2013, 11:48
Post #2





Group: Members
Posts: 822
Joined: 17-September 06
Member No.: 35307



It's always going to be tough to let people look at the spectrum and disguise the MP3. The only way to do so without transcoding the MP3 would probably be to high-pass filter the original (or a freeformat version of it) with a very close-matching opposite filter with zero delay and arithmetically add the waveform containing just the ultrasonics back onto the 320kbps LAME decode or to restrict the test to 320kbps LAME handicapped by having the low-pass disabled. The other approach is to apply the same low-pass filter to the lossless original, which is likely to draw criticisms of its own.

Despite the methodological difficulties, which make the test less definitive and scientifically valid than typical single-encode ABX comparisons, I think the effort may have opened some eyes among audiophiles who simply assume that MP3 is different or worse, so this 'outreach' may have been useful.

I think a 4th option saying "Different, but no idea which is better" would have been problematic in providing a 'get out' for people who instinctively assume MP3 is inferior, so making them pick the better sound unless they said they were identical was necessary to counter that intellectual subconscious bias.

If we consider the Null Hypothesis that "nobody could reliably distinguish A from B" and assume for a moment that it might be true, a statistically significant preference for A over B or B over A might still arise from any number of reasons why people might prefer to choose A over B or B over A (in the same way that people in general asked to choose a random two-digit number that's not easy for the experimenter to guess frequently choose thirty-seven in psychology experiments because it 'feels' more random than most others with repeated, even or consecutive digits or 'round numbers' or those near the middle of the distribution, people might choose A or choose B more frequently)

It might be that a number of replications of your test with the same methodology but a random assignment of A or B to lossless and encoded music might show a similar preference for A over B or B over A even when the audio is swapped, for example.

So the result is not necessarily inconsistent with the Null Hypothesis and due to lack of randomization of A and B between different people's trials, it's not possible to rule out the Null Hypothesis that "no difference could be discerned", nor is it possible to rule out an Alternative Hypothesis that "the encoded file sounds better than the lossless file" or a Second Alternative Hypothesis that "the files sounded different but one was not clearly better than the other - just different"

I certainly think the test has some value and short of programming a server to randomly assign audio to A & B and somehow track that to the participants, it's about the best you can fairly do without ABX or ABC/HR type methodology.

It also has some educational value in experimental design and compromises that may be necessary, so thanks for that.

I think however that the conclusion drawn should be more nuanced than simply stating that the participants preferred MP3 to lossless by an apparently significant margin.

It's an interesting experiment but the correct conclusion to draw isn't obvious in the light of this one test.

Different kinds of tests might have had more power to rule out some of the possible conclusions as false conclusions from the same number of participants, but might not have garnered so many participants.

We're fortunate within HydrogenAudio to have a decent number of participants who are willing to use double-blinded ABC/HR or ABX, which help to eliminate more potential biases or confounding factors and help us to gain reliable conclusions frequently (albeit that it's rare to get a statistically significant result with normal music at medium to high bitrates)
Go to the top of the page
+Quote Post

Posts in this topic
- Archimago   High Bitrate ~320kbps MP3 vs. Lossless audio   Jan 15 2013, 21:11
- - pdq   Just out of curiosity, which forums have you been ...   Jan 15 2013, 21:31
|- - krabapple   QUOTE (pdq @ Jan 15 2013, 15:31) Just out...   Jan 16 2013, 09:02
- - Destroid   QUOTE I've opened up a survey to see if music ...   Jan 16 2013, 12:02
|- - db1989   QUOTE (Destroid @ Jan 16 2013, 11:02) 1. ...   Jan 16 2013, 13:00
- - Destroid   Ok, thanks for catching my rash assumption. So, I...   Jan 16 2013, 13:34
|- - DonP   QUOTE (Destroid @ Jan 16 2013, 07:34) My ...   Jan 16 2013, 14:50
- - Kohlrabi   Reading the test description one flaw seems to be ...   Jan 16 2013, 13:57
|- - DonP   QUOTE (Kohlrabi @ Jan 16 2013, 07:57) Som...   Jan 16 2013, 16:39
||- - Archimago   QUOTE (DonP @ Jan 16 2013, 07:39) QUOTE (...   Jan 20 2013, 01:08
|- - Archimago   QUOTE (Kohlrabi @ Jan 16 2013, 04:57) Rea...   Jan 20 2013, 01:11
- - pdq   There is a further problem. Suppose that a listene...   Jan 16 2013, 14:40
- - 2Bdecided   I'm sure there are examples of 320kbps mp3s be...   Jan 16 2013, 16:00
- - testyou   I'm going to decline your invitation Archimago...   Jan 17 2013, 04:28
|- - Archimago   QUOTE (testyou @ Jan 16 2013, 19:28) I...   Jan 20 2013, 00:57
- - Archimago   FINAL CALL! Closes next week. So far 124 resp...   Jan 25 2013, 21:10
- - Archimago   Study complete! Total respondents - 151. Fol...   Feb 1 2013, 22:39
- - C.R.Helmrich   From your blog: QUOTE This utilizes LAME's ...   Feb 2 2013, 18:07
- - greynol   Wow, I can't believe I once had this topic in ...   Feb 2 2013, 18:14
- - Kohlrabi   I (and others) did question the approach before th...   Feb 2 2013, 20:07
- - greynol   It's pretty obvious the OP registered here onl...   Feb 2 2013, 20:16
- - Archimago   Hang on guys, you're perhaps misinterpreting t...   Feb 3 2013, 07:57
|- - Canar   QUOTE (Archimago @ Feb 2 2013, 22:57) An ...   Feb 3 2013, 08:40
||- - Archimago   QUOTE (Canar @ Feb 2 2013, 23:40) QUOTE (...   Feb 3 2013, 08:54
||- - C.R.Helmrich   Fair enough, given that you apparently didn't ...   Feb 3 2013, 16:01
||- - Archimago   QUOTE (C.R.Helmrich @ Feb 3 2013, 07:01) ...   Feb 3 2013, 17:32
||- - C.R.Helmrich   QUOTE (Archimago @ Feb 3 2013, 17:32) The...   Feb 3 2013, 20:48
||- - Archimago   QUOTE (C.R.Helmrich @ Feb 3 2013, 11:48) ...   Feb 3 2013, 23:56
|- - Kohlrabi   QUOTE (Archimago @ Feb 3 2013, 07:57) Thi...   Feb 3 2013, 16:56
|- - Archimago   QUOTE (Kohlrabi @ Feb 3 2013, 07:56) QUOT...   Feb 3 2013, 17:37
- - db1989   I question whether some factor related to the spec...   Feb 3 2013, 19:05
- - Dynamic   It's always going to be tough to let people lo...   Feb 4 2013, 11:48
- - Archimago   Thanks for the eloquent response Dynamic. Would y...   Feb 5 2013, 01:38


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 1st October 2014 - 01:39