IPB

Welcome Guest ( Log In | Register )

Pre-Test thread, for the 64kbps test
rjamorim
post Aug 22 2003, 07:52
Post #1


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



Hello.

As most of you already know, I am planning to start a 64kbps public listening test in September.

So here are the planned details. Nothing is definitive so far:

The test starts at September 3rd and ends at September 14th

The codecs that will be tested are:
- Ahead HE-AAC "Streaming :: Medium" VBR profile, high quality.
- Ogg Vorbis post-1.0 CVS -q 0
- MP3pro codec in Adobe Audition, VBR quality 35, high quality, m/s and is stereo, no CRC, no narrowing.
- WMAv9 Standard 64kbps (there's no PRO version at such bitrate, AFAIK)
- Real Audio Cook 64kbps (I didn't investigate other settings yet. Comments welcome)

The samples that will be tested have been announced on this thread. If you have concerns/comments about the sample suite choice, please post there.

The test results will be calculated the same way my former tests were. I don't plan to include bitrates in the formula. Comments are welcome now (they are of no use to me after the test has been started).

I haven't decided about anchors yet (my guru is traveling dry.gif), but someone suggested me that I use Lame ABR 128 as higher anchor, so that we can verify which one of these codecs really deliver the marketing of "sounds like MP3 at half the bitrate" biggrin.gif

Then, the lower anchor would be a standard 3.5kHz lowpass, like it is done on most formal tests at low bitrates.

So, I'd like to know your thoughts on what I planned.

Thanks for your attention;

Roberto.

This post has been edited by rjamorim: Aug 22 2003, 07:53


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
 
Start new topic
Replies
Dibrom
post Aug 23 2003, 05:05
Post #2


Founder


Group: Admin
Posts: 2958
Joined: 26-August 02
From: Nottingham, UK
Member No.: 1



QUOTE (Canar @ Aug 22 2003, 08:36 PM)
QUOTE (Dibrom @ Aug 22 2003, 06:44 PM)
As Roberto pointed out earlier, and I agree, I think that adjusting the settings away from the common incarnations, just to "set" the bitrate on this test, calls into question it's credibility.

Not at all. This should give the same result as 2-pass ABR, just done manually, instead of automatically.

I understand that some people do not agree with me. But that is where I stand, and there seem to be people that agree with me. *shrugs*

You seem to have missed the main point of my post:

This test should reflect a real world usage scenario as much as possible.

Artificially tuning the settings does not do this, not to mention that 2-pass ABR doesn't even exist for these codecs (with the exception of WMA maybe? even so, this is not case being called into question).

QUOTE
I do understand where Roberto and co. are coming from and saying that the mode used should be the VBR mode that produces an average of "x" kbps for a broad spectrum of music. The problem is that we're not testing that broad spectrum of music.  We're testing  several individual samples.


This is arguable. We are testing several individual samples comprising a broad spectrum of music. The two conditions are not mutually exclusive. Granted we are not testing every type of music, but that is impossible in any case, and if we're going to nitpick on that point, then we might as well forgo the test entirely.

No matter what the case here, people are going to have to accept that these codecs are not being tested under every condition (genre of music, or even sample for that matter) possible. That means that the results, same as with all tests, have to be taken with a grain of salt, and within context.

QUOTE
In terms of equality, ABR works much better for a small, focused listening test like this one, merely because it provides both a single, standard testing methodology, and uniform bitrates across samples.


I disagree. For one, most ABR modes are based upon VBR (so they share many possible flaws). They are further encumbered though by catering to bitrate first, and quality second (or quality restrained or superceded by bitrate if you prefer).

As I understand it, simply by virtue of emphasing VBR as the method of choice in this test, the point is to measure quality first and bitrate second here. I see no implicit virtue in testing ABR over VBR either -- it would seem to me to be a rather arbitrary choice and one not directly tied to the test's focus which, once again, should be related to provided quality in a real world usage scenario.

Most ABR modes in audio codecs are not entirely accurate in terms of bitrate either, so there is no guarentee that using such a mode will provide any more stable of a foundation than VBR in the first place.

QUOTE
The tester should make every effort to provide as little difference between codecs as possible.


No, the only real necessity is that the tester attempt to equalize the perceived classification of the codecs being tested, only making adjustments where gross mismatches occur.

This means that the absolute bitrate is not nearly as relevant as the wide-scale average bitrate (going beyond samples in this test) or usage scenarios.

What I believe we should be focusing on first is, again, the perceived classification (-q0 vorbis is said to compete with other codec at 64kbps avg mode, for example) first, and the average bitrate second, not worrying about the absolute sample by sample bitrate.

QUOTE
Roberto did not do that with the 128 test. Instead, he threw together a mismatch of VBR and ABR, rationalizing his use of VBR with the broad-spectrum tests.


I cannot speak directly for Roberto, but I believe this is because he was approaching the test from the perspective that I just laid out above.

QUOTE
It remains an inequality between codecs. This is the problem I have with the theory behind these tests.


The problem is that these codecs aren't equal, and so any test including them cannot expect the conditions of comparison to be pefect down to the attribute level of every single codec, only varying by common value. These codecs are different in abilities and behavior, and so if we are to test them and compare them meaningfully and representatively, we need to focus on measuring a generalized and abstracted point (quality levels) not a specific and particular point (performance at exact bitrates).
Go to the top of the page
+Quote Post

Posts in this topic
- rjamorim   Pre-Test thread   Aug 22 2003, 07:52
- - elmar3rd   I wonder if mp3PRO VBR is reliable. FhG fastenc VB...   Aug 22 2003, 09:13
- - tigre   QUOTE (rjamorim @ Aug 21 2003, 10:52 PM)The c...   Aug 22 2003, 09:16
- - tigre   QUOTE (elmar3rd @ Aug 22 2003, 12:13 AM)I won...   Aug 22 2003, 09:22
- - elmar3rd   Beat me, but what about MP3 (LAME or fastenc) at 6...   Aug 22 2003, 09:38
- - askoff   Is it possible to have QuickTime AAC files also in...   Aug 22 2003, 11:17
- - LadFromDownUnder   With regard to the WMA codec (you are correct that...   Aug 22 2003, 11:44
- - rjamorim   QUOTE (elmar3rd @ Aug 22 2003, 05:13 AM)I won...   Aug 22 2003, 16:59
- - bond   thanks for this test rjamorim! as vorbis 1.0....   Aug 22 2003, 17:44
- - tigre   QUOTE (rjamorim @ Aug 22 2003, 07:59 AM)QUOTE...   Aug 22 2003, 17:47
- - rjamorim   QUOTE (bond @ Aug 22 2003, 01:44 PM)as vorbis...   Aug 23 2003, 03:06
- - kotrtim   QUOTE (rjamorim @ Aug 21 2003, 10:52 PM)WMAv9...   Aug 23 2003, 03:43
- - Dibrom   QUOTE (tigre @ Aug 22 2003, 01:16 AM)What kee...   Aug 23 2003, 03:44
- - rjamorim   QUOTE (tigre @ Aug 22 2003, 01:47 PM)I sugges...   Aug 23 2003, 03:48
- - Canar   QUOTE (Dibrom @ Aug 22 2003, 06:44 PM)As Robe...   Aug 23 2003, 04:36
- - wildboar   QUOTE (Canar @ Aug 22 2003, 09:36 PM)The test...   Aug 23 2003, 05:01
- - Dibrom   QUOTE (Canar @ Aug 22 2003, 08:36 PM)QUOTE (D...   Aug 23 2003, 05:05
- - rjamorim   QUOTE (Canar @ Aug 23 2003, 12:36 AM)This see...   Aug 23 2003, 05:09
- - Canar   @wildboar: The multi-pass ABR method I described w...   Aug 23 2003, 06:09
- - rjamorim   QUOTE (Canar @ Aug 23 2003, 02:09 AM)I think ...   Aug 23 2003, 06:17
- - den   QUOTE Atrac3 won't be a good option. It doesn...   Aug 23 2003, 08:33
- - tigre   QUOTE (Dibrom @ Aug 22 2003, 06:44 PM)I don...   Aug 23 2003, 08:38
- - S_O   For Real Gecko Audio you should use the "64 k...   Aug 23 2003, 11:18
- - guruboolez   Some comments : atrac3plus would be nice to test...   Aug 23 2003, 12:54
- - rjamorim   QUOTE (den @ Aug 23 2003, 04:33 AM)Umm sorry,...   Aug 23 2003, 16:17
- - phong   I think a 64kbps test is going to be less tiring t...   Aug 23 2003, 17:45
- - Dologan   Hmm... I frankly don't understand the ABR/CBR/...   Aug 23 2003, 18:40
- - rjamorim   QUOTE (dologan @ Aug 23 2003, 02:40 PM)A sugg...   Aug 24 2003, 01:01
- - guruboolez   Testing eight different files for each sample does...   Aug 24 2003, 01:14
- - rjamorim   That really sounds like a great idea to me. (Alt...   Aug 24 2003, 01:41
- - guruboolez   Is it worth to put two different anchors in this t...   Aug 24 2003, 01:59
- - rjamorim   QUOTE (guruboolez @ Aug 23 2003, 09:59 PM)Is ...   Aug 24 2003, 02:51
- - ErikS   One idea how to add more codecs to the test would ...   Aug 24 2003, 03:56
- - tigre   Would it be an option to use the 8 sliders for tes...   Aug 24 2003, 07:47
- - S_O   QUOTE That's not a real possibility because, o...   Aug 24 2003, 12:14
- - bond   dont think that people will get bored if they test...   Aug 24 2003, 19:37
- - rjamorim   QUOTE (ErikS @ Aug 23 2003, 11:56 PM)One idea...   Aug 24 2003, 20:02
- - rjamorim   QUOTE (S_O @ Aug 24 2003, 08:14 AM)If they do...   Aug 24 2003, 20:10
- - Gecko   I think dividing the test into an essential and a ...   Aug 24 2003, 20:42
- - S_O   QUOTE Well, is HE AAC, WMA, and Real even "cu...   Aug 24 2003, 21:20
- - rjamorim   QUOTE (S_O @ Aug 24 2003, 05:20 PM)But I noti...   Aug 24 2003, 21:31
- - S_O   QUOTE Well, I believe few would allow a stranger t...   Aug 24 2003, 21:57
- - rjamorim   QUOTE (S_O @ Aug 24 2003, 05:57 PM)You donīt ...   Aug 24 2003, 22:23
- - Dologan   Roberto, are you sure you can't merge the resu...   Aug 25 2003, 05:34
- - rjamorim   Well, the biggest problem I see here is using ff12...   Aug 25 2003, 05:47
- - Dologan   Hmm... I see... When are you planning to start th...   Aug 25 2003, 06:34
- - Gabriel   *I think that Guru's idea of 2 sets is interes...   Aug 25 2003, 08:10
- - rjamorim   QUOTE (dologan @ Aug 25 2003, 02:34 AM)When a...   Aug 25 2003, 08:55
- - rjamorim   QUOTE *I think that Guru's idea of 2 sets is i...   Aug 25 2003, 09:01
- - ErikS   * If someone modifies ff123's tool you use for...   Aug 25 2003, 11:17
- - tigre   QUOTE (rjamorim @ Aug 25 2003, 12:01 AM)QUOTE...   Aug 25 2003, 12:16
- - phong   Another problem with using lame as an anchor is th...   Aug 25 2003, 12:48
- - elmar3rd   In statistics, an anchor can be a middle value, no...   Aug 25 2003, 14:48
- - rjamorim   I think we're making a mess out of this test.....   Aug 25 2003, 17:47
- - rjamorim   QUOTE (tigre @ Aug 25 2003, 08:16 AM)A soluti...   Aug 25 2003, 17:52
- - rjamorim   QUOTE (askoff @ Aug 22 2003, 07:17 AM)EDIT: A...   Aug 25 2003, 18:02
- - tigre   QUOTE Well, as already explained somewhere, the An...   Aug 25 2003, 18:28
- - phong   Ok, these are my personal feelings: I think that ...   Aug 25 2003, 18:52
- - rjamorim   QUOTE (phong @ Aug 25 2003, 02:52 PM)I think ...   Aug 25 2003, 20:48
- - Gabriel   Real: I think that it is not needed wma: I think t...   Aug 26 2003, 08:08
- - Digga   QUOTE (Gabriel @ Aug 25 2003, 11:08 PM)If an ...   Aug 26 2003, 09:50
- - Digga   uups, double post...   Aug 26 2003, 09:50
- - LadFromDownUnder   Roberto, given the MS claims about WMA9 (not WMA8)...   Aug 26 2003, 11:06
- - rjamorim   OK, so let's try this: -HE AAC -Vorbis -MP3pr...   Aug 26 2003, 17:36
- - Digga   That combination looks realy good. This way - you...   Aug 26 2003, 19:45
- - elmar3rd   I agree. Of course, 8 codecs will take some time ...   Aug 26 2003, 19:51
- - rjamorim   QUOTE (Digga @ Aug 26 2003, 03:45 PM)I realy ...   Aug 26 2003, 19:57
- - rjamorim   BTW, some more questions that need to be answered:...   Aug 26 2003, 20:02
- - music_man_mpc   QUOTE (rjamorim @ Aug 26 2003, 11:02 AM)IMO, ...   Aug 26 2003, 20:08
- - phong   The list looks good from where I'm standing. ...   Aug 26 2003, 20:22
- - n68   Ciao... read @ the portal. sounds familiar.: *gen...   Aug 26 2003, 20:26
- - rjamorim   QUOTE (n68 @ Aug 26 2003, 04:26 PM)read @ the...   Aug 26 2003, 20:29
- - phong   Huh?   Aug 26 2003, 21:52
- - brett   rj -- as for which aac to include, i would hate to...   Aug 27 2003, 00:59
- - rjamorim   QUOTE (brett @ Aug 26 2003, 08:59 PM)rj -- as...   Aug 27 2003, 01:12
- - Digga   QUOTE i would hate to see you choose ahead just be...   Aug 27 2003, 01:14
- - rjamorim   QUOTE (Digga @ Aug 26 2003, 09:14 PM)I though...   Aug 27 2003, 01:36
- - Gabriel   For mp3-64m I think that blade would be less inter...   Aug 27 2003, 09:04
- - tigre   QUOTE (rjamorim @ Aug 26 2003, 04:12 PM)I don...   Aug 27 2003, 14:41
- - rjamorim   QUOTE (Gabriel @ Aug 27 2003, 05:04 AM)Using ...   Aug 27 2003, 14:50
- - rjamorim   QUOTE (tigre @ Aug 27 2003, 10:41 AM)- Is it ...   Aug 27 2003, 14:58
- - AstralStorm   I'll do a test with ABC/HR on few of samples f...   Aug 27 2003, 15:22
- - rjamorim   QUOTE (AstralStorm @ Aug 27 2003, 11:22 AM)I...   Aug 27 2003, 15:43
- - ErikS   QUOTE (rjamorim @ Aug 27 2003, 02:58 PM)QUOTE...   Aug 27 2003, 15:48
- - rjamorim   QUOTE (ErikS @ Aug 27 2003, 11:48 AM)Que? I w...   Aug 27 2003, 15:51
- - AstralStorm   Rjamorim, you are hereby granted the rights to us...   Aug 27 2003, 15:55
- - AstralStorm   ABC/HR 0.9b doesn't save the results when I tr...   Aug 27 2003, 17:05
- - rjamorim   QUOTE (AstralStorm @ Aug 27 2003, 01:05 PM)Do...   Aug 27 2003, 17:08
- - AstralStorm   Don't forget to enable all additional options ...   Aug 27 2003, 17:21
- - tigre   QUOTE (rjamorim @ Aug 27 2003, 05:58 AM)QUOTE...   Aug 27 2003, 17:34
- - tigre   Results Settings used: <samplename>_CEP_dev...   Aug 28 2003, 01:08
- - tigre   Next result: The Chopin Piano sample. To me surpri...   Aug 28 2003, 12:18
- - Ivan Dimkovic   Hmm.. for plain AAC (not HE AAC) I think that Ahe...   Aug 28 2003, 13:54
- - AstralStorm   Weird... for me LAME Polonaise sample is just unli...   Aug 28 2003, 14:45
- - tigre   QUOTE (Ivan Dimkovic @ Aug 28 2003, 04:54 AM)...   Aug 28 2003, 17:26
- - rjamorim   I guess the biggest problem of lame vs. FhG is for...   Aug 28 2003, 18:15
- - tigre   Not yet. good idea ...   Aug 28 2003, 18:26
- - askoff   rjamorim: Didn't you say that you are looking ...   Aug 29 2003, 11:27
- - tigre   Results Settings used: <samplename>.wav.wa...   Aug 29 2003, 11:37
- - rjamorim   QUOTE (tigre @ Aug 29 2003, 07:37 AM)Conclusi...   Aug 29 2003, 17:24
2 Pages V   1 2 >


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 30th August 2014 - 03:28