Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Multiformat 128 kbps Listening Test (Read 276123 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Multiformat 128 kbps Listening Test

Reply #325 –
I'm glad we are back on topic, and that you took this inniciative as this is something nobody is more qualified at than you.

Regarding the samples. I think we should definitely include some new samples. Maybe it would be a good idea to keep the samples that performed worst and add some new ones.

Multiformat 128 kbps Listening Test

Reply #326 –
Quote
I'm glad we are back on topic, and that you took this inniciative as this is something nobody is more qualified at than you.

Regarding the samples. I think we should definitely include some new samples. Maybe it would be a good idea to keep the samples that performed worst and add some new ones.
[a href="index.php?act=findpost&pid=344037"][{POST_SNAPBACK}][/a]



I think this is a good idea, but I think that one or two of the samples that performed well need to be included for reference.

Multiformat 128 kbps Listening Test

Reply #328 –
Regarding sample selection, I think it would be worth attempting a stab at approximating the bitrate distribution of a large number of samples.  The first step would be to look at the bitrate histograms of the selected codecs to see what's going on over a large sample group.

I'd guess that there would need to be more samples on the low side of the average.  Also, samples with extreme bitrate deviations, both high and low, might be better discarded, as you wouldn't expect to see a lot of them in a small group of random samples.

ff123

BTW, I like a lot of the choices made so far:  only 5 codecs under test, a real low anchor, a defined test objective with rationale for the choices explained (although not universally agreed upon, which is to be expected), and a test administrator who doesn't wither under criticism.

Multiformat 128 kbps Listening Test

Reply #329 –
To sort out one last thing about the encoder (this is related to the samples, too)... Garf asked me if he can send me an "unofficial" encoder if the bug-fixed release of the Nero AAC encoder doesn't make it into the Nero Update before the test starts. Since I cannot wait for him to discuss samples, I have no problem testing an unofficial build as long as I am allowed to distribute it (e.g. make it available for download on HA). The reason why I am "demanding" that is to make sure I don't receive an encoder optimized for the samples used in the test only.

Multiformat 128 kbps Listening Test

Reply #330 –
Nice to see that the test objective is coming together.

1) I think a "yet-to-be-released" encoder version would be nice, if it is also a "will-be-released-soon" encoder.

2) If it won't be released at all (perhaps due to test results), but will again be delayed/fixed, then I think testing it will not be fair and worthwhile.

What do you think?

I understood that the situation is as described in 1). If it's like 2), then I think it's not a good idea to include it.

After all, we are testing for users primarly, not for developers. Right?

Multiformat 128 kbps Listening Test

Reply #331 –
Quote
To sort out one last thing about the encoder (this is related to the samples, too)... Garf asked me if he can send me an "unofficial" encoder if the bug-fixed release of the Nero AAC encoder doesn't make it into the Nero Update before the test starts. Since I cannot wait for him to discuss samples, I have no problem testing an unofficial build as long as I am allowed to distribute it (e.g. make it available for download on HA). The reason why I am "demanding" that is to make sure I don't receive an encoder optimized for the samples used in the test only.
[a href="index.php?act=findpost&pid=344049"][{POST_SNAPBACK}][/a]



This is definitely possible - it is certainly not the intention of nobody inside Nero to generate an encoder which is "produced" for a specific critical sample set.  We never did it, and I am pretty sure we won't do it in the future

Also, new Nero Digital AAC update will prolly bear "ABR" as the recommended codec setting for the 128 kbps range.

Multiformat 128 kbps Listening Test

Reply #332 –
Quote
I think this is a good idea, but I think that one or two of the samples that performed well need to be included for reference.
[a href="index.php?act=findpost&pid=344038"][{POST_SNAPBACK}][/a]

IMO, only really hard to encode samples (not necessarily high-bitrate samples) should be used. If I think of guruboolez' test, where lots of encodings got 4-5 even for his critical ear, and if I think of my own pretty hopeless attempts at ABXing anything besides killer samples at 128kbps with modern encoders, I fear we might get not very discriminative results for the different encoders.
Proverb for Paranoids: "If they can get you asking the wrong questions, they don't have to worry about answers."
-T. Pynchon (Gravity's Rainbow)

Multiformat 128 kbps Listening Test

Reply #333 –
Egor> the link is corrected. Thank you for the verification

ff123> I'm also favourable to your idea. In my opinion, it's the only advantage of changing the samples; otherwise, the "comparison" argument looks strong enough to keep them all.
About extreme samples, I must say that I'm not agree. Especially for low bitrate. As example, I have reconstituted a useful library of LAME VBR encodings to feed my portable players, and IIRC (I can't check now), 10% of the encodings have a really low bitrate (60...100 kbps for 131 kbps on average and 137 as median). And from my experience, all VBR encoder are very far to perform eaqually for the quality point of vue. You maybe remember the Debussy sample tested last year, revealing to some people that different VBR implementations are reacting differently on such musical content. These samples are not exceptional I'd say, and one at least should be included in the test.
The same maybe apply for extremely high bitrate. I'm not concerned, but I suppose that people listening to some kind of electronical music are often encountering such tracks. Again, different encoders don't react eaqually, and this difference would be interested to test (even if one sample is not enough to conclude anything strong for such material).

With 18 samples, including one very low and very high bitrate encoding (assuming that all VBR implementations are excessively reacting to such content) shouldn't be a problem. 16 vacant places for less exceptional content (again, assuming that very low and very high bitrate are exceptional, and I not entirely agree with it) are remaining.
In my own bitrate table, I have such "extreme" samples. Here are the average value for full tracks (bitrate are even more "extreme" with shorter pieces of music)
Code: [Select]
                                 MEAN |      WMA        LAME        iTunes        aoTuV
A03_emese                        178  |     226,4       157,5        143,8        184,0
E01_MODERN_CHAMBER_A_brass        98  |      80,5        78,3        130,2        103,9
E20_MODERN_ORCHESTRAL_E_strings   99  |      63,5        91,5        130,5        112,1
S30_OTHERS_Accordion_A            98  |      74,5        78,9        128,8        108,6
S54_WIND_Trombone_A               98  |      78,0        73,9        129,9        108,9

Only iTunes looks apathetic. I explained why the bitrate don't go below 128 kbps (there's a bitrate floor!). It doesn't go very high either (track A03). But all remaining encoders have similar reactions: either an inflated bitrate or an weedy one.


Sebastian Mares> For the experimental encoder of Nero Digital, I'm a bit puzzled. The test would include on one hand four encoders released for general purpose and widely available to the public and on the other hand one encoder released for the only purpose of the test. I must precise before someone would be tempted to invoke something like a "anti-Nero crusade" that I'd be the first one to be pleased to see very fast improvements of Nero Digital and that I'm in fact currently very keen to see so fast reactions of Ivan and all the Nero Digital team.
I'm just concerned by the equity of the test. Is it fair to accept at the last minute something unofficial and apparently released with the sole object to improve the performance for the test? There were concerns about the inclusion or not of --athaa-sensitivity, which isn't available in latest beta, but what should we say to a whole encoder which isn't available at all at Nero.com and for a majority of Nero users? If we accept this, shouldn't we also ask to Microsoft, Apple, Aoyumi, Xiph, LAME's team if they also can't provide a preview of their upcoming work?

I repeat: I only asking questions, not giving answers. I did my test on my side, and didn't hesitate to use the most advanced encoders (like LAME 3.98 alpha). I never hesitate in fact (I was the only one to test lzst year aoTuV beta 3). In other word, I'm rather in favour of the experimental encoder. But the questions are legitimate in my opinion.

Multiformat 128 kbps Listening Test

Reply #334 –
Quote
Sebastian Mares> For the experimental encoder of Nero Digital, I'm a bit puzzled. The test would include on one hand four encoders released for general purpose and widely available to the public and on the other hand one encoder released for the only purpose of the test.


This is a non-issue,  Nero AAC encoder that will be used in the test will be in the next regular web-release of Nero7.  We had similar cases earleir and all changes ended up in the next public build of N7.
 

Multiformat 128 kbps Listening Test

Reply #335 –
Will the next update of Nero Digital include the tested encoder and produce exactly the same output? If the answer is positive, I agree with you about the non-issue

Multiformat 128 kbps Listening Test

Reply #336 –
It will be identical copy

Multiformat 128 kbps Listening Test

Reply #337 –
Quote
It will be identical copy
[a href="index.php?act=findpost&pid=344141"][{POST_SNAPBACK}][/a]

For me everything is all right then.

Multiformat 128 kbps Listening Test

Reply #338 –
Quote
Quote
To sort out one last thing about the encoder (this is related to the samples, too)... Garf asked me if he can send me an "unofficial" encoder if the bug-fixed release of the Nero AAC encoder doesn't make it into the Nero Update before the test starts. Since I cannot wait for him to discuss samples, I have no problem testing an unofficial build as long as I am allowed to distribute it (e.g. make it available for download on HA). The reason why I am "demanding" that is to make sure I don't receive an encoder optimized for the samples used in the test only.
[a href="index.php?act=findpost&pid=344049"][{POST_SNAPBACK}][/a]



This is definitely possible - it is certainly not the intention of nobody inside Nero to generate an encoder which is "produced" for a specific critical sample set.  We never did it, and I am pretty sure we won't do it in the future

Also, new Nero Digital AAC update will prolly bear "ABR" as the recommended codec setting for the 128 kbps range.
[a href="index.php?act=findpost&pid=344077"][{POST_SNAPBACK}][/a]


It was a just-in-case scenario.  Anyways, I am glad that it's OK with you.

Regarding the samples, I think we could use 12 old samples (from Roberto's listening test) and 6 new ones. I also re-downloaded the samples from my 64 kbps listening test thread and am listening to them now.

Multiformat 128 kbps Listening Test

Reply #339 –
Could anyone post 0:57 - 1:32 2:54 - 3:24 of "Sash! - Stay" (FLAC, WV or MAC)? I hope it's OK even if it's 35 seconds instead of 30...
Edit: I am asking for the long version of the song (~6 minutes) not the radio edit which is only ~3 minutes long.

First 30 seconds of "Sash! - Mysterious Times" would be cool, too.

Multiformat 128 kbps Listening Test

Reply #340 –
I have one comment to target bitrate: Is the goal to have a ~130 kbps bitrate for every file or to achieve this bitrate as average of all samples.

The first possibility would show better how codecs perform at the same bitrate, where the other way would show how can codecs react on more/less complicated sources.

Multiformat 128 kbps Listening Test

Reply #342 –
Quote
I have one comment to target bitrate: Is the goal to have a ~130 kbps bitrate for every file or to achieve this bitrate as average of all samples.

None of the above. The target bitrate is the resulting average bitrate when encoding entire tracks of a wide range of music-genres. You could call it the "real-world average".


Quote
The first possibility would show better how codecs perform at the same bitrate...
[a href="index.php?act=findpost&pid=344343"][{POST_SNAPBACK}][/a]

On a single sample ONLY which is unrealistic and uninteresting - if a sample is difficult, then the encoder is EXPECTED to increase the bitrate *on this part of the track*.

- Lyx
I am arrogant and I can afford it because I deliver.

Multiformat 128 kbps Listening Test

Reply #343 –
I started reading the closed topic in mp3 general forum,
and found there, that MPC is not considered to be tested in this multiformat test due to missing portable hardware support.
This isn't valid, because there is hardware support,
PPC, pda.

Because MPC is an advanced codec, it should be tested also, I think, testing does not hurt.

More reasons to include MPC:
In last listening tests MPC has won or won tied together with Vorbis.
Because MPC has not changed (a lot) during times, MPC could act as a stable anchor, so improvements of other formats during the times in developmnt can be measured.

Then we should look to future, PPC/pda, programmable hardware devices will come more and more, and then you have the possibility to play mpc even more.

(It is already a pleasure to play MPC in my car via  pda, also used for gps navigation.)



regarding mp3 lame settings selection for this test, i agree with gabriel,
that the simple 3.97b1 -V5 --vbr-new should be tested.

Other tests should be carried out in special lame (dev-) tests, like V5 vbr-new vs plain V5, and with/without the ath 1 switch.

Multiformat 128 kbps Listening Test

Reply #344 –
Quote
I have one comment to target bitrate: Is the goal to have a ~130 kbps bitrate for every file or to achieve this bitrate as average of all samples.
[a href="index.php?act=findpost&pid=344343"][{POST_SNAPBACK}][/a]

The same debate happened for most (if not all) previous listening tests involving VBR encoders.
Testing all samples at exactly the same bitrate could be fun, but such test wouldn't correspond to a realistic scenario. I don't know anyone reencoding each track of his library to obtain for each of them a precise bitrate. Unrealistic scenario, but also impossible: such process is only conceivable with encoders offering a precise VBR scale. For iTunes, LAME, Nero Digital, WMA, if the bitrate reach 124 kbps 'only', the use of the next VBR level will lead to 150 or 170 kbps.
That's why such testing principle requires CBR encoders, not VBR ones.

Quote
The first possibility would show better how codecs perform at the same bitrate,
To some extend, the second too. All encoders should lead to the same bitrate. But not on short samples, not on complete track and not even for a complete album -- all encoder should achieve the same bitrate on a full library.
That's the way all people (I suppose) are thinking and acting. --preset standard correspond to something close to 192 kbps, even if the bitrate fluctuate within each track, even if most tracks don't correspond to the expected bitrate. On average, and dispite all dissimilarties, the user will get ~192 kbps with his VBR preset. That's why we can defend the paradoxical idea that all files are going to be tested on the same bitrate even when they aren't obviously encoded at the same bitrate.

Multiformat 128 kbps Listening Test

Reply #345 –
Quote
I started reading the closed topic in mp3 general forum,
and found there, that MPC is not considered to be tested in this multiformat test due to missing portable hardware support.
This isn't valid, because there is hardware support,
PPC, pda.

Because MPC is an advanced codec, it should be tested also, I think, testing does not hurt.

More reasons to include MPC:
In last listening tests MPC has won or won tied together with Vorbis.
Because MPC has not changed (a lot) during times, MPC could act as a stable anchor, so improvements of other formats during the times in developmnt can be measured.

Then we should look to future, PPC/pda, programmable hardware devices will come more and more, and then you have the possibility to play mpc even more.

(It is already a pleasure to play MPC in my car via  pda, also used for gps navigation.)



regarding mp3 lame settings selection for this test, i agree with gabriel,
that the simple 3.97b1 -V5 --vbr-new should be tested.

Other tests should be carried out in special lame (dev-) tests, like V5 vbr-new vs plain V5, and with/without the ath 1 switch.
[a href="index.php?act=findpost&pid=344347"][{POST_SNAPBACK}][/a]


Sorry, but the codecs are now set. I cannot increase the number of codecs tested and IMO, WMA is more popular than MPC so it has priority.
MPC, ATRAC and WMA Pro could be tested in an extension test if people really want it and there are enough testers.

Multiformat 128 kbps Listening Test

Reply #346 –
Quote
I started reading the closed topic in mp3 general forum,
and found there, that MPC is not considered to be tested in this multiformat test due to missing portable hardware support.
This isn't valid, because there is hardware support,
PPC, pda.[a href="index.php?act=findpost&pid=344347"][{POST_SNAPBACK}][/a]

This is not portable players. There are small computers running with OS and allowing the installation of media players and different codec. My Compaq Presario is also portable and play OptimFrog --extrahigh without problem. It's not a valid reason to claim that OptimFrog is supported by portable players
Lets wait that any brand will officially support MPC to say that this encoder corresponds to the required criterion.

Quote
Because MPC is an advanced codec, it should be tested also, I think, testing does not hurt.

Did you read what was decided previously? Advanced is not a criterion of choice. MPC is a pertinent choice, but less than AAC, MP3, Vorbis and WMA.
BTW, an advanced encoder should at least offer a proper seeking 

Quote
Then we should look to future, PPC/pda, programmable hardware devices will come more and more, and then you have the possibility to play mpc even more.

In few years there may be more compatible players with musepack than musepack users themselves 


The debate for MPC, WMAPro, VQF, HE-AAC is over. None of them will be tested here - excepted maybe if we find a very good reason to discard one of the current competitor.

Multiformat 128 kbps Listening Test

Reply #347 –
Quote
The target bitrate is the resulting average bitrate when encoding entire tracks of a wide range of music-genres. You could call it the "real-world average". ...[a href="index.php?act=findpost&pid=344346"][{POST_SNAPBACK}][/a]

I thought this was obvious and already settled.

In my opinion the only possible alternative could be the average bitrate of the complete tracks used for making the samples, but that would not be as representative (as already explained in this thread several times).

BTW, I am just about to post a new table in the "bitrate" thread.


P.S. My WMA findings were commented earlier:
Quote
My results for wma seem to differ quite a bit from Alex b's results. Here, while using the same preset (Q50), the average bitrate is only 98 Kbits/sec. (it seems unfair to compare wma with this preset against the other samples which average 134 Kbits/sec. With wma included the average for all samples drops to 127 Kbits/sec. Unfortunately, if you go to Q75 the bitrate is much too high...)[a href="index.php?act=findpost&pid=343483"][{POST_SNAPBACK}][/a]

I used WMA Pro at q50. I already mentioned earlier that Standard produced about 105 kbps with my test tracks. On the other hand q75 makes way too big files. The only possible "fair" way to use WMA Standard is VBR 2-pass 128 kbps. Though, that is a bit unfortunate because the VBR 2-pass option is not available in the standard WMP10 GUI. It can be found e.g. in the Windows Media Encoder and dbPowerAMP GUIs.

Multiformat 128 kbps Listening Test

Reply #348 –
while it is quite obvious that, in theory, (and mostly in practice too) VBR quality-based should give better results than CBR, I don't think it should be a necessity to include it. The only thing required is: how big is the file. (which translates directly in a maximum bitrate).

What I'm trying to say is: yes, a cbr mode _should_ perform worse than a vbr mode, but it's not a reason to exclude a codec from a test. If a cbr-codec loses out compared to vbr, tough luck. The inverse can also be true: have a look at the results from roberto's tests in which itunes AAC @ CBR performed as well or even better than nero digital @ VBR.

If a file encoded by a particular codec sounds great, I couldn't care less whether it's vbr or cbr...Which mode to use is really the problem of the guy who has to recommend the settings for that encoder.

Multiformat 128 kbps Listening Test

Reply #349 –
Quote
while it is quite obvious that, in theory, (and mostly in practice too) VBR quality-based should give better results than CBR, I don't think it should be a necessity to include it. The only thing required is: how big is the file. (which translates directly in a maximum bitrate).

What I'm trying to say is: yes, a cbr mode _should_ perform worse than a vbr mode, but it's not a reason to exclude a codec from a test. If a cbr-codec loses out compared to vbr, tough luck. The inverse can also be true: have a look at the results from roberto's tests in which itunes AAC @ CBR performed as well or even better than nero digital @ VBR.

If a file encoded by a particular codec sounds great, I couldn't care less whether it's vbr or cbr...Which mode to use is really the problem of the guy who has to recommend the settings for that encoder.[a href="index.php?act=findpost&pid=344363"][{POST_SNAPBACK}][/a]

I don't think any encoders have been excluded because of a missing usable VBR option.

iTunes's default AAC mode is not CBR, the bitrate varies. The new VBR mode makes it vary more, mostly towards upper bitrates.

WMA VBR 2-pass is a good encoding mode at 128 kbps. Most likely it would be better for WMA results than the CBR mode.