IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
Discussion related to multiple format tests' resul, Split from AAC @ 128kbps listening test
duartix
post Feb 11 2004, 20:42
Post #1





Group: Members
Posts: 16
Joined: 11-February 04
Member No.: 11959



Split from here. This topic has been discussed to death in the threads related to the multiformat listening tests.
If you want to add something here, re-read these threads first. Only post here if you have something new to say, please
- tigre

-----------------------------------------------------------------------------------------------------------------------

There is something else that I think would give all codecs a fairer chance. Even though using VBR whenever possible it would be helpfull trying to achieve similar filesizes.
(I remember the multiformat 128kpbs test where Musepack was a winner but at the expense of files 11,7% larger than the close second QT AAC.)

This post has been edited by tigre: Feb 12 2004, 21:06
Go to the top of the page
+Quote Post
schnofler
post Feb 11 2004, 21:49
Post #2


Java ABC/HR developer


Group: Developer
Posts: 175
Joined: 17-September 03
Member No.: 8879



QUOTE (duartix)
I remember the multiformat 128kpbs test where Musepack was a winner

Good thing you said a winner. We wouldn't want Roberto to pass away from banging his head against a wall before the test starts laugh.gif
Go to the top of the page
+Quote Post
music_man_mpc
post Feb 11 2004, 22:02
Post #3





Group: Members (Donating)
Posts: 707
Joined: 20-July 03
From: Canada
Member No.: 7895



QUOTE (duartix @ Feb 11 2004, 11:42 AM)
There is something else that I think would give all codecs a fairer chance. Even though using VBR whenever possible it would be helpfull trying to achieve similar filesizes.
(I remember the multiformat 128kpbs test where Musepack was a winner but at the expense of files 11,7% larger than the close second QT AAC.)

ARGH!! The Musepack files were only 11.whatever percent larger for those samples because those samples were mainly problem samples, so MPC, like any good VBR codec would, allocated more bits than usual! Over a large number of files MPC *at that --quality value* would average 128kbit/s. Do you understand??


--------------------
gentoo ~amd64 + layman | ncmpcpp/mpd | wavpack + vorbis + lame
Go to the top of the page
+Quote Post
rjamorim
post Feb 11 2004, 22:22
Post #4


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (schnofler @ Feb 11 2004, 06:49 PM)
Good thing you said a winner. We wouldn't want Roberto to pass away from banging his head against a wall before the test starts  laugh.gif

Do you see now why I'm retiring in May? wink.gif


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
Continuum
post Feb 11 2004, 22:31
Post #5





Group: Members
Posts: 473
Joined: 7-June 02
Member No.: 2244



Yes, but then the set is not chosen optimally (balanced). The test gave an advantage to highly variable encoding modes. I'm quite sure, that the results would look completely different, if all samples had been low volume classical pieces (which MPC often underbits and which then sometimes sound terrible at this quality setting).

I hope you can find a good compromise between total average bitrate and average bitrate for the sample suite. smile.gif
Go to the top of the page
+Quote Post
music_man_mpc
post Feb 12 2004, 17:01
Post #6





Group: Members (Donating)
Posts: 707
Joined: 20-July 03
From: Canada
Member No.: 7895



QUOTE (Continuum @ Feb 11 2004, 01:31 PM)
I'm quite sure, that the results would look completely different, if all samples had been low volume classical pieces (which MPC often underbits and which then sometimes sound terrible at this quality setting).

This is a mighty hefty claim. Can you post some samples and ABX results please. I normally don't like throwing TOS #8 at people, but with 400+ posts I think that you really should know better.


--------------------
gentoo ~amd64 + layman | ncmpcpp/mpd | wavpack + vorbis + lame
Go to the top of the page
+Quote Post
duartix
post Feb 12 2004, 20:31
Post #7





Group: Members
Posts: 16
Joined: 11-February 04
Member No.: 11959



QUOTE
ARGH!! The Musepack files were only 11.whatever percent larger for those samples because those samples were mainly problem samples, so MPC, like any good VBR codec would, allocated more bits than usual! Over a large number of files MPC *at that --quality value* would average 128kbit/s. Do you understand??

@ music_man_mpc:

Even though my post count may cause you to think otherwise, I'm very familiar with the VBR concept (I've been encoding MPEG4 video since the times of the first Nandub SBC releases). So I will thank you immediately for not regarding my posts as noobarrhea. tongue.gif
Perhaps I'm not so familiar with which audio codecs support VBR. I assumed Quicktime AAC, WMApro & Blade were CBR. But even though I always encode at CQ (constant quality), so always VBR, I guess it's very important for me to know that even though as you correctly have put it:

QUOTE
those samples were mainly problem samples

they were also problem samples for QuicktimeAAC, not just for MPC, BUT it still managed to achieve closely the same quality (overall), even though it used a lower bitrate.

My point is that the day such a codec like Quicktime AAC implements CQ (VBR) encoding, it will use a lower bitrate to achieve the same quality. Clear?
This is a 128kbps test for sound quality, right? So let's isolate the bitrate and test for quality! cool.gif
Otherwise we'll all be encoding at --braindead because we don't need these tiresome ABX tests...

This post has been edited by duartix: Feb 12 2004, 20:38
Go to the top of the page
+Quote Post
music_man_mpc
post Feb 12 2004, 20:46
Post #8





Group: Members (Donating)
Posts: 707
Joined: 20-July 03
From: Canada
Member No.: 7895



QUOTE (duartix @ Feb 12 2004, 11:31 AM)
My point is that the day such a codec like Quicktime AAC implements CQ (VBR) encoding, it will use a lower bitrate to achieve the same quality. Clear?

Ah yes, very clear. Thank you very much for the clairification.

QUOTE (duartix @ Feb 12 2004, 11:31 AM)
This is a 128kbps test for sound quality, right? So let's isolate the bitrate and test for quality!  cool.gif
Otherwise we'll all be encoding at --braindead because we don't need these tiresome ABX tests...


What does this mean, though?


--------------------
gentoo ~amd64 + layman | ncmpcpp/mpd | wavpack + vorbis + lame
Go to the top of the page
+Quote Post
duartix
post Feb 12 2004, 20:52
Post #9





Group: Members
Posts: 16
Joined: 11-February 04
Member No.: 11959



It means that even though we are not as restricted as in "We have to fit 2h of video into 700MB", the lower bitrate to the same quality the better.
Or: the better the quality for the same bitrate the better.
There is a relationship of almost equivalence to these sentences, that is why I'm lobying for the test to resemble the second as close as possible. wink.gif

Cheers,
I'm gone! See you guys tomorrow.
Go to the top of the page
+Quote Post
Continuum
post Feb 12 2004, 22:55
Post #10





Group: Members
Posts: 473
Joined: 7-June 02
Member No.: 2244



QUOTE (music_man_mpc @ Feb 12 2004, 05:01 PM)
QUOTE (Continuum @ Feb 11 2004, 01:31 PM)
I'm quite sure, that the results would look completely different, if all samples had been low volume classical pieces (which MPC often underbits and which then sometimes sound terrible at this quality setting).

This is a mighty hefty claim. Can you post some samples and ABX results please. I normally don't like throwing TOS #8 at people, but with 400+ posts I think that you really should know better.

I didn't know that this was surprising for anyone. The choice of samples has a deep impact on the results. I noticed this effect when I made a small private classical music listening a few months ago, MPC (and Vorbis BTW) had problems with some low volume pieces, more so than Lame. Sadly, I lost my results in a Partition Magic failure, but I can give you at least one example:

Billy Budd.flac (0:25, 1.31 MB): Sample is part of the last track of a recording from 1998 of Benjamin Britten's Billy Budd (under Kent Nagano, singer is Anthony Rolfe Johnson).
I compared
1. Lame 3.90.3 -alt-preset 128. Bitrate is 124 for selection, 125 for whole track,
2. MPC 1.14 --quality 4. Bitrate is 100 for selection, 107 for whole track.

Lame is much better to my ears!
(Out of interest I tried Lame ABR 105 as well. Quality is comparable, but I still find Lame preferable.)

This post has been edited by Continuum: Feb 12 2004, 22:58
Go to the top of the page
+Quote Post
rjamorim
post Feb 12 2004, 23:41
Post #11


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (Continuum @ Feb 12 2004, 07:55 PM)
Sadly, I lost my results in a Partition Magic failure

OT: Partition Magic is the source of all evil.

I (and the German magazine c't) recommend Acronis Partition Expert.


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
guruboolez
post Feb 14 2004, 21:22
Post #12





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (Continuum @ Feb 12 2004, 10:55 PM)
I didn't know that this was surprising for anyone. The choice of samples has a deep impact on the results. I noticed this effect when I made a small private classical music listening a few months ago, MPC (and Vorbis BTW) had problems with some low volume pieces, more so than Lame.

I have already perform similar private tests, and made the results available.
I think that Roberto selected wisely the samples used in tests:
- there are all globaly difficult (that's why average bitrate for VBR encoder is higher than the targetted value), in order to make ABX easier and accessible for most people. Hard to encode, but not total killer.
- the samples are covering a lot of musical genre. That's why "classical" or simply instrumental/vocal music, recorder with hi-fi principle (dynamic range > 5 dB), are only a short part of the sample suit.


Problems with this test is the wrong interpretation of results. You can't answer to a people asking for "the best encoder for classical music at 128 kbps" to look at Roberto's test results. It would be a terrible mistake. First, because individual subjectivity may differ from the overall one; second, because only 2 or 3 samples (on 12) are assimilable to classical music.

But you have a point when you're saying that the sample suit is lacunary. Some good encoders might encounter bad problems with quiet musical samples. I know that MPC have problems at --radio setting with a lot of material, performing better with Metallica latest album than with a Mozart' Symphony. I've already noticed that on extreme cases, MPC --standard have troubles with very very quiet musical parts (removing a lot of noise, details and life in the background). Of course you have to increase the volume in order to perceive the problem, but the missing details are then audible (but is it a problem for most people?).
Go to the top of the page
+Quote Post
Continuum
post Feb 14 2004, 21:52
Post #13





Group: Members
Posts: 473
Joined: 7-June 02
Member No.: 2244



QUOTE (guruboolez @ Feb 14 2004, 09:22 PM)
I think that Roberto selected wisely the samples used in tests:
- there are all globaly difficult (that's why average bitrate for VBR encoder is higher than the targetted value), in order to make ABX easier and accessible for most people. Hard to encode, but not total killer.

That's an interesting thought! Yes, it made the complete test easier. But it also gave a slight edge to more variable codecs (in particular MPC). Maybe, if the files would average to 128, then MPC's rating would be a tad worse with each file?

QUOTE
OT: Partition Magic is the source of all evil.

biggrin.gif Luckily I have more than 10 partitions on my harddisk, and the one affected wasn't the most important one (although it was the largest crying.gif smile.gif).

It was the only problem I ever had with PQMagic, and I used it quite frequently in the last years. I might look at Acronis the next time, thanks for the recommendation.
Go to the top of the page
+Quote Post
guruboolez
post Feb 14 2004, 22:05
Post #14





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (Continuum @ Feb 14 2004, 09:52 PM)
Maybe, if the files would average to 128, then MPC's rating would be a tad worse with each file?

The problem in bitrate calculation: is it meaningul to measure it on very short samples? Isn't it more pertinent to measure the bitrate on (at least) the full track (including quiet passage, silence, difficult and easier part to encode) or (better) the full album?
Go to the top of the page
+Quote Post
Continuum
post Feb 14 2004, 22:13
Post #15





Group: Members
Posts: 473
Joined: 7-June 02
Member No.: 2244



I thought that's how it has been done? The average bitrate was calculated over a large collection of various music styles.
My point is, that the chosen sample suit should be (more) representative of this collection, and contain some "easy" parts as well.

Maybe a low volume piece should be featured in the next test? ph34r.gif
Go to the top of the page
+Quote Post
rjamorim
post Feb 14 2004, 22:23
Post #16


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (Continuum @ Feb 14 2004, 07:13 PM)
Maybe a low volume piece should be featured in the next test? ph34r.gif

It definitely should. But from experience, I know that low volume pieces are too easy for most encoders, so people give up trying to test it and I get very few results for that specific sample. :/


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
guruboolez
post Feb 14 2004, 22:28
Post #17





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (Continuum @ Feb 14 2004, 10:13 PM)
My point is, that the chosen sample suit should be (more) representative of this collection, and contain some "easy" parts as well.

Maybe a low volume piece should be featured in the next test? ph34r.gif

I understand. But I fear that results wouldn't be usable.
I'm maybe wrong, but it seems that Roberto want to throw the "polonaise" sample away because it's too hard for most people to hear a difference. And "polonaise" (piano) is precisely the sample that fit your description (quiet sample, on the first half)...
Go to the top of the page
+Quote Post
rjamorim
post Feb 14 2004, 22:36
Post #18


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (guruboolez @ Feb 14 2004, 07:28 PM)
I'm maybe wrong, but it seems that Roberto want to throw the "polonaise" sample away because it's too hard for most people to hear a difference. And "polonaise" (piano) is precisely the sample that fit your description (quiet sample, on the first half)...

I plan to replace polonaise with Fossiles laugh.gif


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
guruboolez
post Feb 14 2004, 22:45
Post #19





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



I like this sample smile.gif

Did you try this another piano sample? I think there's more problems with this one than with polonaise (strong and sharp attacks).



P.S. In order to answer to you question asked yesterday, I think that the third Symphonie of Mahler (track06) is one of the harder sample of the series. Brass instruments are terrible to reproduce when they're are well-recorded.

For other people, nice classical samples (I used them for my own personal listening tests):
ftp://ftp2.foobar2000.net/foobar/
(link won't be permanent)

This post has been edited by guruboolez: Feb 14 2004, 22:56
Go to the top of the page
+Quote Post
spoon
post Feb 15 2004, 00:19
Post #20


dBpowerAMP developer


Group: Developer (Donating)
Posts: 2757
Joined: 24-March 02
Member No.: 1615



With Roberto's permission I am running background compressions - that is 500 lossless audio files (mixed) are being converted to see how close to 128Kbps the compession comes out at using the VBR AAC codecs. There should be no question of codecs being x% over.


--------------------
Spoon http://www.dbpoweramp.com
Go to the top of the page
+Quote Post
rjamorim
post Feb 15 2004, 00:20
Post #21


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



Yes, thank-you very much, Spoon smile.gif


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 28th November 2014 - 22:37