IPB

Welcome Guest ( Log In | Register )

8 Pages V  « < 3 4 5 6 7 > »   
Reply to this topicStart new topic
128kbps Extension Test - OPEN
spoon
post Jul 25 2003, 09:21
Post #101


dBpowerAMP developer


Group: Developer (Donating)
Posts: 2741
Joined: 24-March 02
Member No.: 1615



After some thought: add up all the average bit rates for each of the samples and lets say for arguement that one codec comes out at 128Kbps and the next 256Kbps, the results of the 256Kbps codec should be scalled back by 50%. Present both sets of results with an explination. If all the Kbps comes within say 5% of 128Kbps then there wouldn't really be a need to adust the figures.


--------------------
Spoon http://www.dbpoweramp.com
Go to the top of the page
+Quote Post
Gabriel
post Jul 25 2003, 09:38
Post #102


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



The currently used test samples are note necessarily representative of overall music. They are chosen test samples. What is important is that the overall bitrate for overall music, using an encoder setting, would be 128kbps on average.

If the average bitrate of those test samples for a specific codec is 180kbps, it is still fair is the overall bitrate for overall music is 128kbps. What a user wants is 128kbps as an overall. A user does not care if 10 seconds of his track are encoded using 250kbps while the next 10 seconds are encoded using 90kbps, as long as overall it is still 128kbps.

regarding vbr codecs vs abr/cbr codecs:
As a Lame developper, I know that Lame in abr mode will be penalized in this test against, as an example, Vorbis which is a vbr codec.
But (unfortunately for Lame) this is perfectly fair as it is representative of codecs abilities.
Lame has currently no high quality vbr mode averaging 128kbps, but this is not a point that should be used to penalize other codes which have such a mode. We should not contrain codecs to use the lowest mode just because some of them are not able to use better modes.
Go to the top of the page
+Quote Post
bawjaws
post Jul 25 2003, 09:55
Post #103





Group: Members
Posts: 173
Joined: 10-December 02
Member No.: 4043



QUOTE (rjamorim @ Jul 24 2003, 05:57 PM)
I added this text to the presentation page:

"I noticed that, on some files, bitrate goes very high, nearing 200kbps sometimes. Is that right?

Yes, and that's the beauty of VBR encoding. It will simply ignore bitrate limitations (whenever possible), using as much bits as needed to encode a problematic sample.
Although that raises issues of fairness, it's the best way to compare modern codecs that shine the most in VBR mode, like Musepack and Vorbis. Trying to force a VBR setting to match as best as possible a desired bitrate, although fairer, is far from the usual practice of audio encoding, where it's more usual that an user just stick to a quality setting, not caring much about a specific bitrate"

Any remarks?

I think the problem some people have with this is they think the VBR codecs are cheating by going higher than 128kbps because they are focusing too tightly on the test alone. You may need to broaden their perspective.

I think it might make sense to people if you explain that the VBR codecs are being rewarded for the savings they make in the easier to encode sections where the 'less intelligent' CBR encoders are wasting bits.

If the bitrate for a given quality level averages 128kbps in ordinary use but is variable then obviously it must go both higher and lower than 128kbps. The reason you don't test on the sections that go lower is because those are the parts that the codec finds easy and so you would be less likely to spot the flaws in the encoder.

Just another angle on this obviously un-intuitive point.

This post has been edited by bawjaws: Jul 25 2003, 09:56
Go to the top of the page
+Quote Post
askoff
post Jul 25 2003, 10:45
Post #104





Group: Members
Posts: 445
Joined: 23-December 02
Member No.: 4214



Does WMA9 include free VBR mode?

This post has been edited by askoff: Jul 25 2003, 10:45
Go to the top of the page
+Quote Post
danchr
post Jul 25 2003, 10:52
Post #105





Group: Members
Posts: 487
Joined: 6-April 03
From: Århus, Denmark
Member No.: 5861



A small pat on the back for Roberto:
A search for "listening test" on Google reveals "Roberto's public listening tests" as the best match.

That means there are plenty of links to it out there wink.gif
Go to the top of the page
+Quote Post
bond
post Jul 25 2003, 12:01
Post #106





Group: Members
Posts: 881
Joined: 11-October 02
Member No.: 3523



QUOTE
I noticed that, on some files, bitrate goes very high, nearing 200kbps sometimes. Is that right?

i wouldnt write it that way as it will make people think that there are many files which are around 200, which isnt the case
i wouldnt write 200kbps also, i would only write:
"I noticed that, on some files, bitrate can go up pretty high. Is that right?"

QUOTE
Yes, and that's the beauty of VBR encoding. It will simply ignore bitrate limitations (whenever possible), using as much bits as needed to encode a problematic sample.

i wouldnt write that, as talking about that is "beautifull" that codecs ignore bitrate goals (whenever possible) isnt really an argument against people who claim that the vbr codecs "are cheating" imho wink.gif

QUOTE
Although that raises issues of fairness, it's the best way to compare modern codecs that shine the most in VBR mode, like Musepack and Vorbis. Trying to force a VBR setting to match as best as possible a desired bitrate, although fairer, is far from the usual practice of audio encoding, where it's more usual that an user just stick to a quality setting, not caring much about a specific bitrate

1) now this sounds like you think for yourself, that the test is unfair, but also that it has to be unfair as the vbr codecs "shine the most in VBR mode". this is also no argument against people claiming imho
2) i wouldnt tell the reader about his behaviour as everybody behaves different (for example i think that the usual practice in audio encoding for the masses is that people care to stay around a specific bitrate on the long run)

QUOTE
quality settings for each codec were chosen because they average to 128kbps over a number of encoded albums. It would be unfair to tie the hands of VBR codecs and punish them for being smart about where to spend what turns out to be the same number of bits over the long run.

now this is the most important part! and the only part which should be mentioned on the homepage imho!
as it says that

the vbr codecs will reach, with this settings, normally the average of 128kbps!!!

that's the argument that counts (no need to talk about fairness or user behaviour or the beauty of vbr codecs)
like what gabriel wrote:

QUOTE (Gabriel @ Jul 25 2003, 10:38 AM)
What is important is that the overall bitrate for overall music, using an encoder setting, would be 128kbps on average.

If the average bitrate of those test samples for a specific codec is 180kbps, it is still fair is the overall bitrate for overall music is 128kbps. What a user wants is 128kbps as an overall. A user does not care if 10 seconds of his track are encoded using 250kbps while the next 10 seconds are encoded using 90kbps, as long as overall it is still 128kbps.


sorry for my bad english

This post has been edited by bond: Jul 25 2003, 12:05


--------------------
I know, that I know nothing (Socrates)
Go to the top of the page
+Quote Post
bond
post Jul 25 2003, 12:02
Post #107





Group: Members
Posts: 881
Joined: 11-October 02
Member No.: 3523



so all in all i would write it that way:

    Q: I noticed that, on some files, bitrate can go up pretty high. Is that right?

    A: Yes, this can happen for samples with hard to encode content, and this is wanted that way as vbr codecs are able to put more bitrate where it is needed and less bitrate where it is not needed that much.

    The settings for each codec were chosen because in the long run they average to 128kbps!

nothing less nothin more (a short but clear statement), no need to talk about 200kbps, about fairness, defending vbr codecs against anything, about how users behave (everybody has different opinions here)...

in the long run every codec setting averages to 128kbps - thats the only important message!

(and i wouldnt put it as question 1, as this is the presentation page where people can read how to participate, i would put it as question 5 (after linux) no need to push people on this info wink.gif )

This post has been edited by bond: Jul 25 2003, 12:12


--------------------
I know, that I know nothing (Socrates)
Go to the top of the page
+Quote Post
ilikedirtthe2nd
post Jul 25 2003, 12:39
Post #108





Group: Members
Posts: 470
Joined: 26-October 01
From: Germany
Member No.: 352



sorry for being off-topic here, but: why were nero/psytel forced to cbr 128 in the prior aac test? the arguments given for this test are also suitable for the aac test. nero -streaming also reaches an average bitrate of 128 on long distance (i suppose).

if you put this into relation, ogg vorbis should have been forced to cbr in this test.

regards; ilikedirt

This post has been edited by ilikedirtthe2nd: Jul 25 2003, 12:42
Go to the top of the page
+Quote Post
AstralStorm
post Jul 25 2003, 12:50
Post #109





Group: Members
Posts: 745
Joined: 22-April 03
From: /dev/null
Member No.: 6130



Heh, this arguing is of no use smile.gif

Rjamorim, have you taken my batch file to hide the results?
You'd need to modify and reupload all packages, but cheating in the test won't be as easy.


--------------------
ruxvilti'a
Go to the top of the page
+Quote Post
ilikedirtthe2nd
post Jul 25 2003, 13:00
Post #110





Group: Members
Posts: 470
Joined: 26-October 01
From: Germany
Member No.: 352



QUOTE (AstralStorm @ Jul 25 2003, 11:50 AM)
Heh, this arguing is of no use smile.gif

i am not arguing against this test here, but maybe we are testing against the wrong aac encoder? maybe psytel -streaming would have won the prior competition if it were allowed to show the beauty of it's vbr algorithms wink.gif

please don't hit me ph34r.gif biggrin.gif
Go to the top of the page
+Quote Post
bond
post Jul 25 2003, 13:06
Post #111





Group: Members
Posts: 881
Joined: 11-October 02
Member No.: 3523



ilikedirtthe2nd
when you read the test again you will find the info that guruboolez (sorry if i misspelled the name) also tested the nero vbr options and came to the conclusion that quicktime is still better!
you'll also find more info about that on the aac test results page http://rarewares.hydrogenaudio.org/test/aa...st/results.html and especially here: http://membres.lycos.fr/guruboolez/AUDIO/a...tableau_vbr.txt

This post has been edited by bond: Jul 25 2003, 13:07


--------------------
I know, that I know nothing (Socrates)
Go to the top of the page
+Quote Post
AstralStorm
post Jul 25 2003, 13:54
Post #112





Group: Members
Posts: 745
Joined: 22-April 03
From: /dev/null
Member No.: 6130



Oh, when I can continue the test not afraid I'll need to redo it?


--------------------
ruxvilti'a
Go to the top of the page
+Quote Post
loophole
post Jul 25 2003, 14:01
Post #113





Group: Members
Posts: 273
Joined: 18-June 03
Member No.: 7254



At the end of the day, if the vorbis encode has an average bit-rate of 190kbps over QuickTime or WMA's encode of 128kbps, that's still 62kbps extra it's allocating itself which is almost 50% extra of what the original bit-rate was targeted to be. Vorbis has managed bitrates, it should be forced to use it - after all, LAME was.

If you're against forcing Vorbis to use managed bitrates, maybe you should have encoded them at a given quality level (say they averaged 190kbps), then used the 192kbps setting for the ABR codecs. Seems fairer to me.

Also, Compressor which comes with Final Cut Pro 4 seems to have VBR options for AAC, not just ABR.

(for those who think QuickTime Pro is CBR, it isn't - it's ABR)

It just does sound unfair to the ABR codecs to me. It seems to me like you're testing the accuracy of the bitrate engine, not which codec sounds better for any given bitrate.

Taken to the extreme, codec x in VBR mode y could allocate whatever it wants, and end up an average of 300kbps but that would be a fair comparison because it's engine was smart enough to allocate more bits. </SARCASM>
Go to the top of the page
+Quote Post
AstralStorm
post Jul 25 2003, 14:17
Post #114





Group: Members
Posts: 745
Joined: 22-April 03
From: /dev/null
Member No.: 6130



We're testing quality of settings averaging 128kbps on multiple different albums.
Not on short samples. That's all. LAME and Quicktime are using ABR,
because they either don't have VBR modes or VBR modes perform worse than ABR.


--------------------
ruxvilti'a
Go to the top of the page
+Quote Post
JohnV
post Jul 25 2003, 14:18
Post #115





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



QUOTE (loophole @ Jul 25 2003, 04:01 PM)
At the end of the day, if the vorbis encode has an average bit-rate of 190kbps over QuickTime or WMA's encode of 128kbps, that's still 62kbps extra it's allocating itself which is almost 50% extra of what the original bit-rate was targeted to be.

How do you know that it's 50% exta of what the original bitrate was targeted to be, you are testing just short hard-to-encode clips. The targeted quality setting tested produces 128kbps average, and we are testing the quality of specific quality setting of a vbr codec which gives this average bitrate. We are not testing qualities of 12 different quality settings of one vbr codec in one test.

QUOTE
Vorbis has managed bitrates, it should be forced to use it - after all, LAME was.
Like both Gabriel and me have said in this thread already, so this is prolly 4th time in this thread alone: There's no high quality low bitrate Lame vbr setting. ABR mode in Lame averaging 128kbps performs better.


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
ff123
post Jul 25 2003, 14:28
Post #116


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



QUOTE (loophole @ Jul 25 2003, 05:01 AM)
At the end of the day, if the vorbis encode has an average bit-rate of 190kbps over QuickTime or WMA's encode of 128kbps, that's still 62kbps extra it's allocating itself which is almost 50% extra of what the original bit-rate was targeted to be. Vorbis has managed bitrates, it should be forced to use it - after all, LAME was.

Lame uses ABR because it's generally agreed that ABR sounds best at bitrates which average about 128 kbit/s. Vorbis uses -q (VBR) because that's the mode it generally sounds best using.

QUOTE
If you're against forcing Vorbis to use managed bitrates, maybe you should have encoded them at a given quality level (say they averaged 190kbps), then used the 192kbps setting for the ABR codecs. Seems fairer to me.


If one uses the Vorbis quality mode, and for a particular sample it averages 190, then the ABR codecs shouldn't be bumped up to 192 for that sample -- that's exactly the same thing as forcing the VBR codecs down to 128, except in reverse!

QUOTE
Also, Compressor which comes with Final Cut Pro 4 seems to have VBR options for AAC, not just ABR.

(for those who think QuickTime Pro is CBR, it isn't - it's ABR)


The purpose of the last test was to find the AAC codec to be used for this test. Final Cut Pro 4 wasn't tested.

QUOTE
It just does sound unfair to the ABR codecs to me. It seems to me like you're testing the accuracy of the bitrate engine, not which codec sounds better for any given bitrate.

Taken to the extreme, codec x in VBR mode y could allocate whatever it wants, and end up an average of 300kbps but that would be a fair comparison because it's engine was smart enough to allocate more bits. </SARCASM>


The VBR codecs in this test are quite stable in average bitrate on a per-album basis. If a VBR codec can manage to produce 128 kbit/s on average, but ends up with 300 kbit/s on a difficult sample, it should not be penalized for doing its job! That's why VBR exists. The caveat to this is that the VBR codecs should not sound worse on "easy" samples.

ff123
Go to the top of the page
+Quote Post
upNorth
post Jul 25 2003, 14:53
Post #117





Group: Members
Posts: 1099
Joined: 18-March 03
From: Oslo, Norway
Member No.: 5569



This reminds me of discussions about "optimized" command lines for encoding, that people that don't know the first thing about compression come up with. Especially the ones that "improve" the -presets. How come people trust the developers (and the people helping them) to develop the codec, but not at all when it comes to using it. It makes no sense to me and I hope this test will be performed as intended and not in an artificial and "fair" way. Everything always seem very easy when you don't know nothing about it...
QUOTE (rjamorim)
I did the test. To the best of my knowledge and capacity. If you dislike it, either run your own test or f*ck off.
B)wub.gif biggrin.gif
Thanks for arranging this test and keep up the good work smile.gif

edit: typo

This post has been edited by upNorth: Jul 25 2003, 14:54
Go to the top of the page
+Quote Post
ff123
post Jul 25 2003, 15:46
Post #118


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



I've completed 9 samples, and I'm tracking my preferences:

http://ff123.net/friedman/stats.html

(Blocked ANOVA analysis)

The losers don't surprise me, but my apparent preference does (although it's not clearcut by any means).

ff123
Go to the top of the page
+Quote Post
rjamorim
post Jul 25 2003, 16:29
Post #119


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (upNorth @ Jul 25 2003, 10:53 AM)
Thanks for arranging this test and keep up the good work  smile.gif

Thanks smile.gif


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
atici
post Jul 25 2003, 18:05
Post #120





Group: Members (Donating)
Posts: 1180
Joined: 21-February 02
From: Chicago
Member No.: 1367



Maybe ATRAC3 LP2 mode could've been added to this test too. And we could have referred to the results every once in a while MiniDisc zealot shows up in HA wink.gif


--------------------
The object of mankind lies in its highest individuals.
One must have chaos in oneself to be able to give birth to a dancing star.
Go to the top of the page
+Quote Post
Gecko
post Jul 25 2003, 18:19
Post #121





Group: Members
Posts: 937
Joined: 15-December 01
From: Germany
Member No.: 662



QUOTE (askoff @ Jul 25 2003, 10:45 AM)
Does WMA9 include free VBR mode?

After reading this thread, that's what I thought. If you use 2 pass you are forcing the encoder to a specific bitrate. If WMA9 Pro had a proper VBR implementation, it would maybe have increased the bitrate on critical samples as well.

This thread shows one thing: correct testing of VBR codecs, especially when mixed with cbr/managed bitrate ones, is all but trivial. Maybe this should be emphasized on the main page and then quickly explain why this method was chosen.

Here's an idea for future tests. Instead of using regular difficult test samples, you would use samples that all come out at ca the same bitrate. That would make the testfield more leveled and you are testing what a codec can do at a certain bitrate without tweaking.

But of course it will be tedious to find samples that will produce nearly identical bitrates over a range of codecs and you will probably end up with samples that are easy to encode and harder to identify by ear.

This post has been edited by Gecko: Jul 25 2003, 18:20
Go to the top of the page
+Quote Post
rjamorim
post Jul 25 2003, 18:30
Post #122


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (Gecko @ Jul 25 2003, 02:19 PM)
and you will probably end up with samples that are easy to encode and harder to identify by ear.

That's exactly the biggest issue.

QUOTE
Maybe ATRAC3 LP2 mode could've been added to this test too. And we could have referred to the results every once in a while MiniDisc zealot shows up in HA


Problem is, I already had my share of bad issues with SonicStage (nearly fuxored my Win2k installation), so I'll only test Atrac3 once I create a VirtualPC partition only for installing and running it. :-P


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
atici
post Jul 25 2003, 18:31
Post #123





Group: Members (Donating)
Posts: 1180
Joined: 21-February 02
From: Chicago
Member No.: 1367



QUOTE
Here's an idea for future tests. Instead of using regular difficult test samples, you would use samples that all come out at ca the same bitrate.


I haven't read in detail how this test is performed. But in general isn't it a better idea for testing different codecs to fix a bitrate and adjust the quality level (for mpc and vorbis) so that the output is going to be exactly equal to that bitrate? This way every codec will be given the same amount of space to demonstrate their skills. Let's say 128kbps lame, q4.3 MPC, q4.5 vorbis for a specific sample, but 128kbps lame, q5 mpc, q5.5 vorbis for another sample...

This post has been edited by atici: Jul 25 2003, 18:32


--------------------
The object of mankind lies in its highest individuals.
One must have chaos in oneself to be able to give birth to a dancing star.
Go to the top of the page
+Quote Post
rjamorim
post Jul 25 2003, 18:38
Post #124


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (atici @ Jul 25 2003, 02:31 PM)
I haven't read in detail how this test is performed. But in general isn't it a better idea for testing different codecs to fix a bitrate and adjust the quality level (for mpc and vorbis) so that the output is going to be exactly equal to that bitrate? This way every codec will be given the same amount of space to demonstrate their skills. Let's say 128kbps lame, q4.3 MPC, q4.5 vorbis for a specific sample, but 128kbps lame, q5 mpc, q5.5 vorbis for another sample...

JohnV will shoot you


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
JohnV
post Jul 25 2003, 18:44
Post #125





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



Does anybody here read what ff123, Gabriel, Guruboolez and I are writing in this very thread at least 5 times each so far? I can't BELIEVE THIS!! headbang.gif



--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post

8 Pages V  « < 3 4 5 6 7 > » 
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 29th July 2014 - 19:26