IPB

Welcome Guest ( Log In | Register )

Why is MPC perceived to be the best?, (an off-topic audio encoding discussion)
ScorLibran
post Feb 8 2004, 18:43
Post #1





Group: Banned
Posts: 769
Joined: 1-July 03
Member No.: 7495



This is a question that has floated through my mind for most of a year, but only yesterday became more clear to me.

Actually, it's a two-part question...

... 1. Is MPC commonly accepted among the HA community as the best psychoacoustic encoding format? (i.e., the most efficient at achieving perceptual transparency.)

... 2. If so, why?

The first item I've heard stated quite frequently, but have never seen any results of "transparency threshold tests" that would reveal the superior efficiency of MPC. I've heard that MPC uses superior encoding technology, but I'm referring more to the end result of such development efforts...the perceived sound quality, as measured against other codecs at the point of perceptual transparency for a significant number of people.

These concerns on my part were born from a post I made here, where the points of MPC statistically tying other formats at 128kbps, but no other known test results existing, were brought up. The thread portion ended up in the recycle bin, but I'm taking the chance that my concerns about calling MPC "the best" weren't the reason it was put there.

Hence, I want to bring up this idea in a different context in the off-topic forum (in the hope that this will be the correct area for it).

What I'd like to see, for instance, for the education of myself and others, would be a results summary like the following (though this is a very simplistic example)...

Format............Perceptual Transparency Threshold (nominal bitrate across samples tested)
MPC.................nnn kbps
AAC.................nnn kbps
Vorbis..............nnn kbps

...and so forth

Granted, VBR is more efficient at mid-bitrates and up, and quality-based VBR modes aren't bitrate centric, but we need some means of measurement and comparison between codecs in this context, so if not calling it "nominal bitrate", then perhaps "average filesize per minute of audio across all samples"

Perceptual Transparency Threshold could have a fixed target, like >90% samples with 5.0 subjective ratings, and non-differentiable from reference with ABX testing.

This kind of test has been discussed before, and has been mostly viewed as having little "real-world value". And I agree. Roberto's tests are much more relevant for most music listeners, and for determining the best formats for useful purposes like streaming audio, portable players, etc.

Many of us (including me) have trouble testing even these bitrate ranges, so higher ones would be even more tedious, and would answer not as many pressing questions.

My point, though, is how can MPC be called "the best for achieving transparency" without a test such as this? (Because so far it's been shown to be only "among the best" at lower rates.)
Go to the top of the page
+Quote Post
 
Start new topic
Replies
ScorLibran
post Feb 9 2004, 07:36
Post #2





Group: Banned
Posts: 769
Joined: 1-July 03
Member No.: 7495



QUOTE (rjamorim @ Feb 8 2004, 10:07 PM)
QUOTE (ScorLibran @ Feb 8 2004, 07:52 PM)
If nothing "formal" was ever done, I'd still like to see some kind of initiative to provide evidence that "MPC is the best" if it's going to claimed as such.

May I suggest you conduce such test? wink.gif

I would give as much support as I can.

Well, that's an offer I can't refuse! wink.gif

Unless anyone else would want to run one sooner, I'll have (or make) time this coming spring to do this.

As mentioned, my goal is to see, and not just for MPC, where the average transparency threshold for each of the top five codec "families" resides, based on a variety of samples and variety of listeners (as many as possible...this is critical). ABX on every sample would be required for this test, which would make it more tedious, unfortunately. And using a rating scale would have less meaning, and may not even be needed. The goal is not to compare how "good" each codec sounds compared to the reference, but rather simply find the "point of threshold" for each one on each sample, then average results across samples and then across participants.

The same statistical analysis as Roberto uses should be adhered to to make the results as meaningful as possible.

This would not be a bitrate-centric test, but rather an attempt to "slowly turn up the quality dial", so to speak, until the participant can no longer ABX at p<0.05. We obviously have to start low enough that a significant number of people can actually ABX samples with that p-value or lower. Then try (as) gradually (as possible) increasing bitrates until they can't ABX them with confidence. This would be done with each format in the test.

So, everyone please post whether you think...

1. this test would be possible/manageable,
2. with enough participation, results from this test would be meaningful enough to justify it, and
3. the best method and approach that could balance manageability and effectiveness.


A few preliminaries of how I'd like to do it (or see it done)...
  • Five formats: LAME MP3, WMA Pro, MPC, AAC, Vorbis. These are the most discussed formats in HA, and the most used in the world, I think. Once variant/version of each. Which variant of AAC (and possibly Vorbis) should be used may require a pre-test.
  • At least 3 encoding rates/quality settings. I'd like to see more than 2 to be able to represent (at least minimally) what could be considered a "range" of encoding rates with each format. Target lower and upper range limits would probably be 128kbps and 192kbps, based on enough sub-4 ratings on Roberto's 128kbps Extension Test, and on the fact that many people say they prefer settings which generally yield rates that seem to center around 192kbps (160-224kbps) for maximum efficiency. (Speaking of which...providing evidence to support people's beliefs about "maximum efficiency" would be exactly the point of this test.)
  • 8 samples should be sufficient to cover enough musical variety to make the results meaningful for most people, I would hope, while keeping the numbers down in the interest of manageability.
  • ABX testing, with a target p-value of 0.05 or less, for each sample. Each encode rate (by format) would be double-blind tested to find not it's "level of transparency", but rather which side of the "threshold of transparency" it's on for each particular tester.
  • Results would be gathered for each tester, each format, each encode rate, and each sample. Averages would be compiled by sample, then across encode rates, then across participants, then the end-numbers for each format would be graphed with statistical error margins shown. (Since there would be no subjective aspect to this test, the error margins could perhaps instead compensate for a limited number of participants, to make the results meaningful for a majority of people.)
If all goes well, we could show an average perceptual transparency threshold for the music samples tested. And something recent to point to when someone asks "where does MPC/MP3/Vorbis/WMA-Pro/AAC become transparent" or "which codec is the most efficient at mid-high bitrates"? (Since Roberto has done so much to provide the same info in other bitrate ranges.)


Issues/Questions/Brainstorming:
  • 5 formats x 3 encoding rates x 8 samples = 120 test groups. Is this feasible? If not, we could break this up into a seperate test for each format, or maybe into 2 "phases"...2 formats in one phase and 3 in the other. Or maybe ideally, have one test per format, and a more discreet scale...perhaps 5 encode rates? This would stretch the schedule, though, allowing at least 15 days (+ or -) per format plus time to have pre- and post-test discussions, determine samples, compile results at the end, etc.
  • How can people be convinced to participate in a listening test that would be more tedious than most previous tests, when some of those tests even had trouble getting enough participants themselves?


This post has been edited by ScorLibran: Feb 9 2004, 07:38
Go to the top of the page
+Quote Post

Posts in this topic
- ScorLibran   Why is MPC perceived to be the best?   Feb 8 2004, 18:43
- - sthayashi   1) The answer appears to be yes. 2) This result ca...   Feb 8 2004, 19:53
- - Canar   There's never been a hard and fast test as to ...   Feb 8 2004, 21:51
- - ScorLibran   Thanks for the input, guys. If nothing "f...   Feb 8 2004, 22:52
- - Canar   The smartest way to do it would be to encode the p...   Feb 8 2004, 23:13
- - ddrawley   The proof of MPC quality is neither vague nor simp...   Feb 9 2004, 03:38
- - Eli   Its more than the quality of the codec that draws ...   Feb 9 2004, 03:45
- - rjamorim   QUOTE (ScorLibran @ Feb 8 2004, 07:52 PM)If n...   Feb 9 2004, 04:07
- - indybrett   This is what I would like to see compared. It woul...   Feb 9 2004, 05:09
- - ScorLibran   QUOTE (rjamorim @ Feb 8 2004, 10:07 PM)QUOTE ...   Feb 9 2004, 07:36
- - Kalamity   Looks like double-nested QUOTES puts the auto-quot...   Feb 9 2004, 09:00
- - 2Bdecided   Just how many people are going to give you anythin...   Feb 9 2004, 12:51
- - Der_Iltis   Is there any site you would suggest where i can fi...   Feb 9 2004, 17:04
- - bubka   QUOTE (indybrett @ Feb 8 2004, 11:09 PM)This ...   Feb 9 2004, 17:08
- - sthayashi   QUOTE (ScorLibran @ Feb 9 2004, 01:36 AM)How ...   Feb 9 2004, 17:47
- - music_man_mpc   This problem has bothered me as well in the past. ...   Feb 9 2004, 21:30
- - MGuti   if all the samples were problem samples, this woul...   Feb 9 2004, 22:00
- - ChristianHJW   To make this test sensible, you have to remove the...   Feb 9 2004, 22:03
- - Continuum   Some remarks: I think neither MP3 nor Vorbis is a...   Feb 9 2004, 22:08
- - music_man_mpc   QUOTE (Continuum @ Feb 9 2004, 01:08 PM)QUOTE...   Feb 9 2004, 22:19
- - Kalamity   QUOTE (music_man_mpc)We should start making some p...   Feb 10 2004, 00:07
- - music_man_mpc   QUOTE (Kalamity @ Feb 9 2004, 03:07 PM)QUOTE ...   Feb 10 2004, 01:16
- - Kalamity   QUOTE (music_man_mpc)I agree with you in terms of ...   Feb 10 2004, 01:52
- - ScorLibran   QUOTE (Kalamity @ Feb 9 2004, 03:00 AM)Some o...   Feb 10 2004, 02:08
- - music_man_mpc   QUOTE (Kalamity @ Feb 9 2004, 04:52 PM)QUOTE ...   Feb 10 2004, 02:18
- - ScorLibran   QUOTE (Kalamity @ Feb 9 2004, 07:52 PM)I do n...   Feb 10 2004, 02:30
- - Doctor   For narrowing down quality settings use binary sea...   Feb 10 2004, 02:43
- - Kalamity   QUOTE (ScorLibran)QUOTE (Kalamity @ Feb 9 200...   Feb 10 2004, 02:59
- - Vertigo   Do we ask why god is omnipotent? He just is....sa...   Feb 10 2004, 04:00
- - rjamorim   QUOTE (Vertigo @ Feb 10 2004, 01:00 AM)Do we ...   Feb 10 2004, 04:07
- - Dologan   The god part or the MPC part?   Feb 10 2004, 04:22
- - Mr_Rabid_Teddybear   QUOTE (rjamorim @ Feb 9 2004, 07:07 PM)QUOTE ...   Feb 10 2004, 04:33
- - music_man_mpc   QUOTE (Doctor @ Feb 9 2004, 05:43 PM)For narr...   Feb 10 2004, 04:59
- - music_man_mpc   QUOTE (music_man_mpc @ Feb 9 2004, 07:59 PM)E...   Feb 10 2004, 15:43
- - Doctor   QUOTE (music_man_mpc @ Feb 10 2004, 09:43 AM)...   Feb 11 2004, 02:17
- - music_man_mpc   QUOTE (Doctor @ Feb 10 2004, 05:17 PM)Regardi...   Feb 11 2004, 04:53
- - Eli   Wouldnt a problem sample set make the most sense? ...   Feb 11 2004, 04:55
- - music_man_mpc   QUOTE (Eli @ Feb 10 2004, 07:55 PM)Wouldnt a ...   Feb 11 2004, 05:00
- - ScorLibran   Well, the time frame for this test should really b...   Feb 11 2004, 05:24
- - Kalamity   QUOTE (music_man_mpc @ Feb 10 2004, 07:53 PM)...   Feb 11 2004, 06:22
- - SometimesWarrior   Whoa, I haven't posted here in 6 months! I...   Feb 11 2004, 06:41
- - ChangFest   QUOTE Here's my point: if we're testing fo...   Feb 11 2004, 16:18
- - Eli   QUOTE (music_man_mpc @ Feb 10 2004, 11:00 PM)...   Feb 11 2004, 16:57
- - ChangFest   QUOTE However if one codec handles problem samples...   Feb 11 2004, 23:04
- - Eli   QUOTE (ChangFest @ Feb 11 2004, 05:04 PM)QUOT...   Feb 11 2004, 23:16
- - SometimesWarrior   QUOTE (Eli @ Feb 11 2004, 02:16 PM)IMHO a tes...   Feb 12 2004, 00:38
- - Doctor   Hm, I feel a flamefest waiting to happen. Before e...   Feb 12 2004, 00:59
- - Doctor   Concerning the goal of the test. It is obviously i...   Feb 12 2004, 01:10
- - Kalamity   I would not consider this thread flammable, let al...   Feb 12 2004, 05:23
- - 2Bdecided   Before spending hours discussing and thinking abou...   Feb 12 2004, 12:18
- - Continuum   QUOTE (2Bdecided @ Feb 12 2004, 12:18 PM)Cons...   Feb 12 2004, 12:40
- - 2Bdecided   QUOTE (Continuum @ Feb 12 2004, 11:40 AM)QUOT...   Feb 12 2004, 12:51
- - fanerman91   This "Really Big Codec Test" sounds exci...   Apr 18 2004, 04:18
- - ScorLibran   QUOTE (fanerman91 @ Apr 17 2004, 10:18 PM)Thi...   Apr 18 2004, 05:43
- - damiandimitri   QUOTE Consider each negative ABX result as a ...   Apr 19 2004, 12:04
- - tigre   QUOTE (damiandimitri @ Apr 19 2004, 01:04 PM)...   Apr 19 2004, 12:46


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 26th November 2014 - 03:01