IPB

Welcome Guest ( Log In | Register )

2 Pages V  < 1 2  
Reply to this topicStart new topic
High Bitrate ~320kbps MP3 vs. Lossless audio, Spoiler: mp3s were TRANSCODED from Lame free-format.
Kohlrabi
post Feb 3 2013, 16:56
Post #26





Group: Super Moderator
Posts: 1047
Joined: 12-March 05
From: Kiel, Germany
Member No.: 20561



QUOTE (Archimago @ Feb 3 2013, 07:57) *
This was TRUE LOSSLESS vs. DOUBLE MP3 (free-form LAME) encoding so I could retain the full spectrum and have the MP3 set NOT look like the standard "easy to identify" MP3 spectrum.

Of course people could look at the spectrum while testing, there's no way I can control that EXCEPT by masking it with turning off the lowpass filter, but I could not do that and have the spectrum "look right" unless I went slightly above 320kbps - hence the double pass so I could get the "error" within the 320kbps level.
I will be glad the day when everybody accepts that MP3 encoding is more than just a lowpass. By obsessing over the lowpass of LAME you destroyed everything else that makes LAME a good lossy encoder, like the psychoacoustic model.

QUOTE (Archimago @ Feb 3 2013, 07:57) *
Even with the double LAME compression passes, people still thought the MP3 sounded better than lossless! As the report shows, those using >$6000 gear got it even more wrong than those using <$500.
I always assumed those overpaying audiofools had bad hearing, thanks for showing at least that. biggrin.gif

QUOTE (Archimago @ Feb 3 2013, 07:57) *
Is it because those owning >$6000 gear [...] had bad hearing?
Well, this might be the only real outcome of this test, if any.

This post has been edited by Kohlrabi: Feb 3 2013, 16:56


--------------------
Ceterum censeo Masterdiskem esse delendam.
Go to the top of the page
+Quote Post
Archimago
post Feb 3 2013, 17:32
Post #27





Group: Members
Posts: 13
Joined: 22-December 12
Member No.: 105342



QUOTE (C.R.Helmrich @ Feb 3 2013, 07:01) *
Fair enough, given that you apparently didn't know about ABC-HR and given that the "result says no difference" issue pdq first mentioned did not occur. But the bottom line is: your listeners heard a difference between A and B. Proponents of lossy (but transparent) coding can now argue, "of course they heard a difference, the MP3 encoding was badly done", while opponents might argue, "this test used some enhanced (free-format, not low-passed) 350-kbps MP3 and doesn't tell you anything about whether listeners would have preferred real-life non-tandem-coded MP3". See what I mean?

QUOTE
This was TRUE LOSSLESS vs. DOUBLE MP3 (free-form LAME) encoding so I could retain the full spectrum and have the MP3 set NOT look like the standard "easy to identify" MP3 spectrum.

I think we all understood this perfectly. The problem is that when you have to modify a lossy encoder to hide what it's doing from your test participants, your test is flawed.

Chris


Yes, Helmrich, I do fully understand this and talked about the "unorthodox" encoding and how it could affect the audio quality in a negative way. The hope in the test was to demonstrate that lossy encoding does not deserve a "bad rap" as has been generally portrayed and I think this has been achieved when I look at the response from the audiophile crowd.

This post has been edited by Archimago: Feb 3 2013, 17:33
Go to the top of the page
+Quote Post
Archimago
post Feb 3 2013, 17:37
Post #28





Group: Members
Posts: 13
Joined: 22-December 12
Member No.: 105342



QUOTE (Kohlrabi @ Feb 3 2013, 07:56) *
QUOTE (Archimago @ Feb 3 2013, 07:57) *
This was TRUE LOSSLESS vs. DOUBLE MP3 (free-form LAME) encoding so I could retain the full spectrum and have the MP3 set NOT look like the standard "easy to identify" MP3 spectrum.

Of course people could look at the spectrum while testing, there's no way I can control that EXCEPT by masking it with turning off the lowpass filter, but I could not do that and have the spectrum "look right" unless I went slightly above 320kbps - hence the double pass so I could get the "error" within the 320kbps level.
I will be glad the day when everybody accepts that MP3 encoding is more than just a lowpass. By obsessing over the lowpass of LAME you destroyed everything else that makes LAME a good lossy encoder, like the psychoacoustic model.

QUOTE (Archimago @ Feb 3 2013, 07:57) *
Even with the double LAME compression passes, people still thought the MP3 sounded better than lossless! As the report shows, those using >$6000 gear got it even more wrong than those using <$500.
I always assumed those overpaying audiofools had bad hearing, thanks for showing at least that. biggrin.gif

QUOTE (Archimago @ Feb 3 2013, 07:57) *
Is it because those owning >$6000 gear [...] had bad hearing?
Well, this might be the only real outcome of this test, if any.


I did not say that MP3 was somehow just lowpass. Lowpass is just the "pathognomonic" sign of MP3 encoding; the nasty fingerprint by which anyone who has an interest in looking at the spectrum would detect. Knowing that in an open survey like this where people are free to look, this somehow must be hidden using the MP3 encoder itself.

This post has been edited by Archimago: Feb 3 2013, 17:39
Go to the top of the page
+Quote Post
db1989
post Feb 3 2013, 19:05
Post #29





Group: Super Moderator
Posts: 5275
Joined: 23-June 06
Member No.: 32180



I question whether some factor related to the specific processing methodology might be responsible for the skew of listeners in favour of the encoded files. One would more logically expect an approximately 50/50 split between lossless and lossy when the latter is at a generally transparent bitrate such as 320 kbps. The fact that 52% thought MP3 sounded better, whereas only 18% perceived no difference, suggests either some effect of your double conversion that is making the MP3s sound more appealing in some way or simply really bad and somewhat arbitrary judgements by the relevant respondents.
Go to the top of the page
+Quote Post
C.R.Helmrich
post Feb 3 2013, 20:48
Post #30





Group: Developer
Posts: 686
Joined: 6-December 08
From: Erlangen Germany
Member No.: 64012



QUOTE (Archimago @ Feb 3 2013, 17:32) *
The hope in the test was to demonstrate that lossy encoding does not deserve a "bad rap" as has been generally portrayed and I think this has been achieved when I look at the response from the audiophile crowd.

Alright. Once some (positive or negative) criticism from that audiophile crowd appeared, it would be nice if you could summarize it or link to it in this thread. I'd be curious to know whether you convinced some long-term sceptics. And please, call me Chris smile.gif

Chris


--------------------
If I don't reply to your reply, it means I agree with you.
Go to the top of the page
+Quote Post
Archimago
post Feb 3 2013, 23:56
Post #31





Group: Members
Posts: 13
Joined: 22-December 12
Member No.: 105342



QUOTE (C.R.Helmrich @ Feb 3 2013, 11:48) *
QUOTE (Archimago @ Feb 3 2013, 17:32) *
The hope in the test was to demonstrate that lossy encoding does not deserve a "bad rap" as has been generally portrayed and I think this has been achieved when I look at the response from the audiophile crowd.

Alright. Once some (positive or negative) criticism from that audiophile crowd appeared, it would be nice if you could summarize it or link to it in this thread. I'd be curious to know whether you convinced some long-term sceptics. And please, call me Chris smile.gif

Chris


Hi Chris,
I don't know if I have time ahead to to summarize in great detail the response from audiophiles... Since this test started on AudioAsylum in the "PC Audio" subforum, it might be worthwhile following the thread there.

Nonetheless I have already received E-mail such as the little snip below...

"This test was VERY VERY educating. I write this not because I am in the
wrong-guessers camp, but because it demonstrated to me the following:
1. My very minor exposure to mp3 material had led me to assume that mp3
sounds BAD without actual listening to it in my life - less than several
dozen times and through smallish desktop speakers.
Truth is mp3 can sound very good."

He goes on to describe tests he did himself with the LAME procedure I used and his further thoughts... Great to see that this test has catalyzed some consideration on the matter.

This test was never aimed at scientific rigor (although I have tried to document and describe the procedure as best I can for full disclosure and critique); but rather allow questioning of a certain commonly held sentiment. Like the response above shows, it has at least affected some people who I suspect would never in their life sit down to try a properly controlled ABX.

This post has been edited by Archimago: Feb 4 2013, 00:03
Go to the top of the page
+Quote Post
Dynamic
post Feb 4 2013, 11:48
Post #32





Group: Members
Posts: 819
Joined: 17-September 06
Member No.: 35307



It's always going to be tough to let people look at the spectrum and disguise the MP3. The only way to do so without transcoding the MP3 would probably be to high-pass filter the original (or a freeformat version of it) with a very close-matching opposite filter with zero delay and arithmetically add the waveform containing just the ultrasonics back onto the 320kbps LAME decode or to restrict the test to 320kbps LAME handicapped by having the low-pass disabled. The other approach is to apply the same low-pass filter to the lossless original, which is likely to draw criticisms of its own.

Despite the methodological difficulties, which make the test less definitive and scientifically valid than typical single-encode ABX comparisons, I think the effort may have opened some eyes among audiophiles who simply assume that MP3 is different or worse, so this 'outreach' may have been useful.

I think a 4th option saying "Different, but no idea which is better" would have been problematic in providing a 'get out' for people who instinctively assume MP3 is inferior, so making them pick the better sound unless they said they were identical was necessary to counter that intellectual subconscious bias.

If we consider the Null Hypothesis that "nobody could reliably distinguish A from B" and assume for a moment that it might be true, a statistically significant preference for A over B or B over A might still arise from any number of reasons why people might prefer to choose A over B or B over A (in the same way that people in general asked to choose a random two-digit number that's not easy for the experimenter to guess frequently choose thirty-seven in psychology experiments because it 'feels' more random than most others with repeated, even or consecutive digits or 'round numbers' or those near the middle of the distribution, people might choose A or choose B more frequently)

It might be that a number of replications of your test with the same methodology but a random assignment of A or B to lossless and encoded music might show a similar preference for A over B or B over A even when the audio is swapped, for example.

So the result is not necessarily inconsistent with the Null Hypothesis and due to lack of randomization of A and B between different people's trials, it's not possible to rule out the Null Hypothesis that "no difference could be discerned", nor is it possible to rule out an Alternative Hypothesis that "the encoded file sounds better than the lossless file" or a Second Alternative Hypothesis that "the files sounded different but one was not clearly better than the other - just different"

I certainly think the test has some value and short of programming a server to randomly assign audio to A & B and somehow track that to the participants, it's about the best you can fairly do without ABX or ABC/HR type methodology.

It also has some educational value in experimental design and compromises that may be necessary, so thanks for that.

I think however that the conclusion drawn should be more nuanced than simply stating that the participants preferred MP3 to lossless by an apparently significant margin.

It's an interesting experiment but the correct conclusion to draw isn't obvious in the light of this one test.

Different kinds of tests might have had more power to rule out some of the possible conclusions as false conclusions from the same number of participants, but might not have garnered so many participants.

We're fortunate within HydrogenAudio to have a decent number of participants who are willing to use double-blinded ABC/HR or ABX, which help to eliminate more potential biases or confounding factors and help us to gain reliable conclusions frequently (albeit that it's rare to get a statistically significant result with normal music at medium to high bitrates)
Go to the top of the page
+Quote Post
Archimago
post Feb 5 2013, 01:38
Post #33





Group: Members
Posts: 13
Joined: 22-December 12
Member No.: 105342



Thanks for the eloquent response Dynamic.

Would you be OK if I took this message and added it to the "comments" on the blog page? As you alluded to, the experimental design is absolutely important and I certainly want to make sure readers get a chance to review what you just said.

This post has been edited by Archimago: Feb 5 2013, 01:38
Go to the top of the page
+Quote Post

2 Pages V  < 1 2
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 18th September 2014 - 04:22