Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: 128kbps Extension Test - OPEN (Read 56887 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

128kbps Extension Test - OPEN

Reply #75 –
Quote
Still have my priviledge

Quote
Now, tell me what can you say overall about the average 128kbps quality (certain quality setting) based on results from those? Or actually any useful result.. I'd like to know...


The reason VBR exists is that a codec can be advanced enough to lower its bitrate and up its bit rate, but for this to be a fair test - especially with different sample types - for all we know on the harpsicord: ogg will go to an average of 200Kbps, whilst WMA goes to 128Kbps. Now you could say, tough luck WMA for not matching Ogg when it goes to 200Kbps, but you could also say that WMA is better programmed because it stays within its quality range and does not vary wildly. *** these codecs and numbers are totally made up ***

I am thinking then, if the codec has ABR avaliable it should be used in preferrence to VBR in this type of test.

In testing different vbr codecs nothing is ideal, but the current method is hell of lot better than tweaking short hard-to-encode samples forcefully to 128kbps and then try to make some statistical conclusions about that vbr codec's quality which should potray in someway the average quality relative to other codecs.
Juha Laaksonheimo

128kbps Extension Test - OPEN

Reply #76 –
Quote
I am thinking then, if the codec has ABR avaliable it should be used in preferrence to VBR in this type of test.

Nice idea, to test Vorbis ABR, completely untweaked (I suppose), completely unused (I'm quite sure), slower and probably worse quality...

Or how doing a listening quality test by forcing the best encoders to be worse than they are...


EDIT :


PS : it's mabe time to split this thread. I'm a bit sad to see whole Roberto's work contested here. It's his test, he worked for it - we worked too for him. It's isn't time anymore to contest the methodology. There were a PRE-TEST DISCUSSION for it.

128kbps Extension Test - OPEN

Reply #77 –
Quote
Reread my post, please.
Modifying the batches won't help - how Roberto will know which file is which?

It has to be set earlier.
Additionally, encoding/decoding order should be 'randomized' (in the batch files, of course).

Sorry, I thought you were talking about a batch file to fix the comma bug.  As far as making the encoding process silent, I wonder if it's worthwhile to insert that in at this point.  What's the point of randomizing the encoding process?

ff123

128kbps Extension Test - OPEN

Reply #78 –
The 128Kbps title of this test is slightly missleading, we will stick with the Harpsichord...for sake of argument if Ogg went to a final Average of 200Kbps whilst all the other Codecs were 130Kbps, it would be fair to say that Ogg would 'win' that one, but what are you telling people? if you had 10 samples to be tested then oggs result would IMHO be 1/10th higher than it should, unless the whole world listens to harpsichord music, which I don't think it does


Edit: Indeed, please split the thread, I have the upmost respect for Roberto

128kbps Extension Test - OPEN

Reply #79 –
one time the vbr codecs are discriminated and the other time the abr codecs (the old thing with apples and oranges)...

but i wouldnt know what to think if mpc will be voted on 1-3 place 

let's wait for the results
I know, that I know nothing (Socrates)

128kbps Extension Test - OPEN

Reply #80 –
Quote
I am thinking then, if the codec has ABR avaliable it should be used in preferrence to VBR in this type of test.

I suppose the reason ABR was not used where applicable is because not all codecs do ABR. Hence, VBR was used since all the formats tested are able to encode in it.

128kbps Extension Test - OPEN

Reply #81 –
Discussions like this are always necessary for the interpretation.

128kbps Extension Test - OPEN

Reply #82 –
I already finished three samples, with quite suprising results for me, for example that a encoder that is inperceptable for me at one file completly sucks at another.
But I have a problem now: The files at audiocoding.com are not downloadable.
They are not down, but server does not respond.
Because I have unlimited traffic and 10MB free, very fast, space I uploaded the first file. If you don´t have it yet and audiocoding doesn´t work for you, too, download here:
Sample01.zip (41_30sec) (6,93MB)
It would be great if someone else could upload sample02, 03 and 04.
Quote
Hence, VBR was used since all the formats tested are able to encode in it.
Also Quicktime AAC? I thought it´s CBR only.

128kbps Extension Test - OPEN

Reply #83 –
I had some problems too with the first four packages last hours. My download software can now DL them. Problem seems to be corrected.

128kbps Extension Test - OPEN

Reply #84 –
@guru, JohnV and others

I appreciate your response, but please don't just curse and link to threads I have already read multiple times. It achieves nothing.

I completely understand the logic behind the choice of codecs settings etc, and I completely understand the use of quality settings in the real world, etc. You should credit the average member of HA with a bit more intelligence. 

I also feel that Roberto has done an excellent job organising a complex public test like this one. My query about bitrate was in no way a reflection on his outstanding efforts and attention to detail.

But at the same time, here we are promoting this as a public, open test, slashdotting here, putting other notices up there, but when the results are interpreted, to many readers the results will be bogus because of the huge descrepancy in the bit rates for a test titles "128k Extension Test". 197k is not 128k in any one's language.

OK, so we should proceed and see how the results turn out, but I can just see that if mpc gets up as the best encoder for example, the vorbis zealots will have an absolute field day attacking the validity of the results, and vice versa.

There are some ways that this concern can be alleviated. First, place a statement clearly explaining that the codecs are VBR, and for some samples, some codecs will exceed 128 k by some margin. Make sure that it is publicly known that the samples are not all the same size. Also, as part of the test results, show the average bitrate for each codec across all the samples, so that at least when people read through the results, they can conclude that a certain encoder may be better for their own needs, but it uses additional bitrate as required, etc. so at least the bitrate bloat for some of these encoders at this setting is not hidden.

Den.

128kbps Extension Test - OPEN

Reply #85 –
In the ideal, here is how I think a VBR test suite would be constructed:

There would be a mix of difficult and easy samples, and one encoding setting is used for the entire test suite such that when the average bitrate was calculated for all the samples, it worked out to 128 kbit/s.  Of course, this would be after verifying that multiple album encodes yielded 128 kbit/s on average too, also using that same encoding setting.  Also, we would do the "slicing" the sample out after encoding an entire song/album routine.  I think this would get rid of some of the criticism we're seeing here.

Real world:

Probably to achieve the average bitrate of 128 kbit/s, we would have to encode many more easy samples than difficult samples, and the test suite would grow very large while not being significantly better at telling us which codec is better (the idea is that difficult samples are best for discriminating quality differences).

The assumption being made is that the relative codec ranking stays the same with the easy samples as it does with the difficult samples, or at least that all the codecs are transparent if the samples are easy to encode.

But I think that making this assumption is far more reasonable than limiting the VBR codecs to 128 kbit/s by changing the encoding settings per sample.  That's not how the codecs are used in real life, and it's nullifying the purpose of using VBR.


Regarding slicing:

The problem with slicing out samples after encoding is that we'd have to assemble another test suite, where the entire songs are available.  This is actually possible to do, though, so it's just a problem of practicality for this test.

ff123

P.S.  I agree that this whole discussion must be made very clear in the results page, because if there is argument and criticism here, it will be ten times worse on other forums.

128kbps Extension Test - OPEN

Reply #86 –
Quote
Why mppenc 1.14 and not 1.15r? As far as I know, it has been very well tested.

Well, I was under the impression that using alphas wouldn't be very safe.

Quote
Ummm, I think we have a problem! 


The joys of VBR...

Quote
Maybe Roberto should put the explanation to his listening test page as well.


Thanks for the idea. I'll do that now

Quote
call it something like hydrogenaudio 128kbps audio codec comparison test


Blah. And since when this is HydrogenAudio's test?

Sorry about being anal here, but I'd rather not have my tests directly connected to any community.

Quote
due to the ","/"." failure in oggenc all .ogg files were encoded with "-q 4" (and not with "-q 4,25")


I don't deserve this >_<

Quote
Ouch.. this is definitely a problem, although not catastrophic. -q4 is officially 128kbps nominal anyway.


Right. Quality shouldn't be much worse either.

Quote
I uploaded modified oggenc.exe that will use proper quality level, it uses recent CVS libraries.


Thanks a lot, Case.

I'll add it to the abc-hr_bin package and reupload it. No point in asking people to retake the test though, IMO.

So, it now just accepts either dot or comma as separator?

128kbps Extension Test - OPEN

Reply #87 –
Quote
I think it should be "post 1.0 CVS".


True. Fixed that on the page.

Quote
some one here who has access to the presentation page to update it with this (not unimportant) info?


Just FYI, people with access to the rarewares account are John33, Dibrom, Ruairi (rc55) and me. You can PM one of them if you need something to be corrected urgently while I'm away.


About the oggenc comma issue:

People, it's only 0.25 (or 0,25  ) difference in the encoded file. I can't be arsed to believe 2-3kbps will make a huge difference.

And as JohnV pointed out, I'm using nominal 128 anyway.


The updated abc-hr_bin package is being uploaded already though.

128kbps Extension Test - OPEN

Reply #88 –
Quote
were is rjamorim

I spent the whole day with my GF. It was about time I gave her some attention, didn't do that since I returned from my trip.

Quote
or does anbody else has access to upload an updated package?


One of them is located at Audiocoding. Menno has access to that account.

The other one is being hosted by Paul Harris (verloren). Also, John33 has access to that account.

128kbps Extension Test - OPEN

Reply #89 –
Quote
Just FYI, people with access to the rarewares account are John33, Dibrom, Ruairi (rc55) and me. You can PM one of them if you need something to be corrected urgently while I'm away.

*cough*  I happen to have access too..
Juha Laaksonheimo

128kbps Extension Test - OPEN

Reply #90 –
Quote
So, it now just accepts either dot or comma as separator?

This fix isn't as advanced as the one I gave to oggenc author when the issue was originally found some months ago, I now just removed locale changing for the encoder and it only accepts dot.

128kbps Extension Test - OPEN

Reply #91 –
I just got started.    Poor blade, it's like a grizzly bear trying to do ballet.  :x  I hope nobody comes to my apartment and sees me locked in the bathroom with extension cords running under the door.  Unfortunately I can still hear the cheetah in my Windoze box seeking, even through the wall.  >_<

I agree an explanation about the VBR up front (before results are compiled) will be key to appeasing the skeptics.  All the codecs used settings that produce an average of about 128kbps over a large number of albums (and you've got the data to back that up.)  VBR codecs can't be punished for spending those bits where they'll do the most good.  Unfortunately, with the festering hole Slashdot has become, there will still be plenty of people who won't accept the results no matter how carefully the test is conducted.  The recent thread on the AAC test was almost entirely flaming by people who either didn't read or didn't understand the test.
I am *expanding!*  It is so much *squishy* to *smell* you!  *Campers* are the best!  I have *anticipation* and then what?  Better parties in *the middle* for sure.
http://www.phong.org/

128kbps Extension Test - OPEN

Reply #92 –
@JohnV: Sorry, I forgot you got root access.

Quote
But I have a problem now: The files at audiocoding.com are not downloadable.
They are not down, but server does not respond.


Audiocoding is actually sourceforge. Their servers aren't very reliable, but I would be a bastard if I complained about something I get for free. Please, just try again later.

Quote
OK, so we should proceed and see how the results turn out, but I can just see that if mpc gets up as the best encoder for example, the vorbis zealots will have an absolute field day attacking the validity of the results, and vice versa.


Ok, here's my take on the "I think your test is wrong" I expect some critics to come up with. As you guys know, I'm a very blunt person

[span style='font-size:12pt;line-height:100%']I did the test. To the best of my knowledge and capacity. If you dislike it, either run your own test or f*ck off.[/span]

Of course, I accept constructive criticism. But I will really ask people that come to me saying my test sucks without providing a better way to do it to go fuck themselves. I actually learned to be blunt from my experiences with the AAC test. :B

Quote
There are some ways that this concern can be alleviated. First, place a statement clearly explaining that the codecs are VBR, and for some samples, some codecs will exceed 128 k by some margin.


Almost finished

128kbps Extension Test - OPEN

Reply #93 –
Quote
Ok, here's my take on the "I think your test is wrong" I expect some critics to come up with. As you guys know, I'm a very blunt person

I did the test. To the best of my knowledge and capacity. If you dislike it, either run your own test or f*ck off.


I was about to post, saying thank you for responding to my initial concern a little more calmly than some others... 

Quote
Almost finished 


Excellent. I think that should do it. Time to get on with the testing.  B)

Thanks again Roberto.

Den.

128kbps Extension Test - OPEN

Reply #94 –
I added this text to the presentation page:

"I noticed that, on some files, bitrate goes very high, nearing 200kbps sometimes. Is that right?

Yes, and that's the beauty of VBR encoding. It will simply ignore bitrate limitations (whenever possible), using as much bits as needed to encode a problematic sample.
Although that raises issues of fairness, it's the best way to compare modern codecs that shine the most in VBR mode, like Musepack and Vorbis. Trying to force a VBR setting to match as best as possible a desired bitrate, although fairer, is far from the usual practice of audio encoding, where it's more usual that an user just stick to a quality setting, not caring much about a specific bitrate"

Any remarks?

128kbps Extension Test - OPEN

Reply #95 –
Quote
I was about to post, saying thank you for responding to my initial concern a little more calmly than some others...

Well, that wasn't targeted at you. I was just giving people a sample of what awaits clueless critics.




edit: Just uploaded new abc-hr_bin.zip packages using Case's oggenc (thanks!), all should work as planned now. :B

128kbps Extension Test - OPEN

Reply #96 –
Quote
Any remarks?


Sweet. Explains the situation without apologising for it.  B)

I know it's early days yet, but some comments about the same, and reported bitrates would be useful when the test results are published too, but I'm sure you're already onto that!

Quote
Well, that wasn't targeted at you. I was just giving people a sample of what awaits clueless critics.


 

Den.

128kbps Extension Test - OPEN

Reply #97 –
Quote
Any remarks?

I would add that the quality settings chosen for the VBR codecs were chosen because they average out to about 128kbps over a number of encoded albums.  And to pick a nit, I might say "fairer in one sense" instead of just "fairer' and would add that it would be unfair to tie the hands of VBR codecs and punish them for being smart about where to spend what turns out to be the same number of bits over the long run.

BTW: Let me also say thank you for running this test.  You rule.
I am *expanding!*  It is so much *squishy* to *smell* you!  *Campers* are the best!  I have *anticipation* and then what?  Better parties in *the middle* for sure.
http://www.phong.org/

128kbps Extension Test - OPEN

Reply #98 –
Added your comments (actually, did some copy-pasting  hope you don't mind).
Thanks a lot for your remarks, and your kind words.

128kbps Extension Test - OPEN

Reply #99 –
About this VBR issue: a similar thing happened with the 64 kbps test. Some people said that that was not the way of testing codecs, but it was just because they didn't understand the issue.

Test designers are not responsible for people being dumb. You can explain why VBR codecs have been used this way. If some people still don't agree with explanation, it's their problem. The only "but" I can accept is that for the type of music someone listens, the average VBR bitrate is higher than 128 kbps. People complaining should provide their VBR average bitrate to support their complaints.
Even when that can happen in some particular cases, it's a particular problem impossible to overcome with a general test, which is still much better than nothing.