IPB

Welcome Guest ( Log In | Register )

3 Pages V  < 1 2 3 >  
Reply to this topicStart new topic
Personal multiformat listening test at ~130 kbps, based on classical (baroque) music only
JohnV
post Oct 13 2003, 13:53
Post #26





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



QUOTE (guruboolez @ Oct 13 2003, 03:42 PM)
I know that with other musical genre, Q90 gives higher bitrate. That's mainy why I choose ABR 128 for the whole listening test: no bitrate deviation. Roberto made the same choice for his 128 public test: Q75 is closer to 128 kbps with most music, but too low with classical samples.

Imo you should have used Q75... If it gives too low bitrate on classical music, that's the codec's problem. You didn't tweak the other codecs' vbr quality levels either, rather used what generally gives about 128kbps with large selection of material (well, Nero streaming LC is somewhat lower; 110-120 on average).


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
guruboolez
post Oct 13 2003, 14:05
Post #27





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (JohnV @ Oct 13 2003, 01:53 PM)
QUOTE (guruboolez @ Oct 13 2003, 03:42 PM)
I know that with other musical genre, Q90 gives higher bitrate. That's mainy why I choose ABR 128 for the whole listening test: no bitrate deviation. Roberto made the same choice for his 128 public test: Q75 is closer to 128 kbps with most music, but too low with classical samples.

Imo you should have used Q75... If it gives too low bitrate on classical music, that's the codec's problem. You didn't tweak the other codecs' vbr quality levels either, rather used what generally gives about 128kbps with large selection of material (well, Nero streaming is somewhat lower; 110-120 on average).

Yes, I have : I choosed --preset 134 for the same reason.
Don't forget that I did this test for classical only, and some practice purpose. I'm not going to rate Q75, and compare it to ABR 128, if average bitrate is near 90 kbps with my music.
If I had test some metal, or jazz gallery, Q75 would probably be my choice. Here, Q90 seemed to be more accurate.

N.B. Nero STREAMING gives higher bitrate with classical samples than with metal one. It's really close to 130 kbps. With some pure strings discs, average bitrate is superior to 140 kbps (not 100% sure though).

Only exception is mpc : --radio gives too high bitrate here, compared to others file format settings.
Go to the top of the page
+Quote Post
JohnV
post Oct 13 2003, 14:27
Post #28





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



QUOTE (guruboolez @ Oct 13 2003, 04:05 PM)
Yes, I have : I choosed --preset 134 for the same reason.
Don't forget that I did this test for classical only, and some practice purpose. I'm not going to rate Q75, and compare it to ABR 128, if average bitrate is near 90 kbps with my music.
If I had test some metal, or jazz gallery, Q75 would probably be my choice. Here, Q90 seemed to be more accurate.

N.B. Nero STREAMING gives higher bitrate with classical samples than with metal one. It's really close to 130 kbps. With some pure strings discs, average bitrate is superior to 140 kbps (not 100% sure though).

Only exception is mpc : --radio gives too high bitrate here, compared to others file format settings.

Right. Problem here is that you use relatively higher vbr quality level for one codec, based on how it acts with your music. You don't need to be a prophet to guess that wma9 pro q90 does pretty good, if it's average with vast selection of material is close to 170kbps (I can't say for sure is it). Q90 is pretty high quality level. When you set a vbr quality level, you set (simplified) a certain level of how much (audible) quantization noise is allowed in codec's opinion (but the point here is that psychoacoustics is often wrong, and if you artifically pump up the bitrate, you aren't getting very useful data generally, only for very specific case).
But if you encode other than classical music, you need to use q75 to be close to 128kbps. Your test doesn't say anything about Q75 which is more closer to 128kbps with non-classical and general music (all mixed). I hope people understand that these results are very very restricted on very specific cases. I fear most don't understand.

Ps. also don't look at the Nero Streaming bitrates of any development version yet.
Based on your file sizes, the Nero release codec you used stays clearly under 128kbps.

Also because vbr codecs do act differently, imo the only usable result to say anything about some codec's true quality on a certain quality level, is to use vast selection of material and use the average bitrate from that as indicator.


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
spoon
post Oct 13 2003, 15:04
Post #29


dBpowerAMP developer


Group: Developer (Donating)
Posts: 2745
Joined: 24-March 02
Member No.: 1615



It matters not how it performs quality wise on other music, this test is for classical only so q90 was a fair setting.

If you want to do yor own test on a different type of music then please do, don't rubbish the results of this test.


--------------------
Spoon http://www.dbpoweramp.com
Go to the top of the page
+Quote Post
ScorLibran
post Oct 13 2003, 15:45
Post #30





Group: Banned
Posts: 769
Joined: 1-July 03
Member No.: 7495



IMHO, I think it actually *is* a real world scenario for some people to select incremental -q settings to target certain nominal bitrates for certain music types. For me, Vorbis (1.0) -q 4.00 averaged ~120kbps on a cross-section of my music, while -q 4.25 more closely hit my desired average of 128kbps (it actually averages ~129kbps) in the interest of getting my music encoded with as many bits as possible while not causing buffering problems on my more limited playback hardware. Other people may have other, comparable reasons, but I wouldn't venture whether this would be a majority or not. However, this would be one reason for a normal user to "focus" a codec's nominal bitrate for their musical taste by using not-so-general quality settings.

Also, I personally see the value in having both types of listening tests...the codec tests using a wide variety of music types as in Roberto's tests, and also "style-specific" tests, such as the one guruboolez has performed here. I've noted people saying (in this very thread as a matter of fact) that different codecs perform differently with different types of music, so that's clear justification to have "style-specific tests" to compliment the "general tests". And accordingly, quality settings / nominal bitrates should be chosen for the style-specific tests to make sure that each codec is throwing an equal number of bits at each sample, as closely as possible anyway.

If q75 "generally" provides 128kbps for a codec on most types of music, then that would be fine if you're testing "most types" of music. This was a test on Baroque-era classical, so crippling one codec by using a -q setting because it gives the desired-target bitrate with rock is not fair nor realistic because rock's not being tested. And for people who only listen to classical and not any rock, what nominal bitrate rock music hits with the codecs being tested will mean nothing to them.

By "focusing" each codec's -q setting for the specific type of music (based on the music's characteristics), you are measuring (I would think) more precisely how that codec uses the (approximate) same number of bits to encode the music vs. each other codec. It levels the playing field, so to speak. If a codec's q75 gives a significantly lower nominal bitrate (whether it's "supposed to" or not) with the type of music being tested than the other codecs give, then consider that a person who listens to this type of music (and not much else) would like to know which codec provides the best performance when each codec is generating very similar bitrates and filesizes with their music.

My $0.02.

This post has been edited by ScorLibran: Oct 13 2003, 15:46
Go to the top of the page
+Quote Post
rjamorim
post Oct 13 2003, 15:59
Post #31


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



I also agree with guruboolez' methodology. He's not trying to extrapolate results to other musical styles, he's testiong only one style. For that reason, adapting quality settings is acceptable.

Also, keep in mind that if he used q75, he would be flooded with whiner posts shouting "APPLES AND ORANGES! APPLES AND ORANGES!"


@guruboolez: Thanks a lot for another very interesting listening test. You're a legend wink.gif


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
guruboolez
post Oct 13 2003, 16:14
Post #32





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



CODE
Based on your file sizes, the Nero release codec you used stays clearly under 128kbps.


I'm not sure. I didn't have exact average bitrate, only filesize :

mp3 abr 134 : 7 615 221
QT AAC cbr 128 : 7 856 071
WMA 128 : 7 673 606
Nero : 7 537 148

Maybe under 128 kbps, but really close to others encoding.
The problem with Nero AAC ont his kind of music : bitrate is often higher than with other musical genre, and with probably a worse quality than with other genres...


QUOTE
I hope people understand that these results are very very restricted on very specific cases. I fear most don't understand.


I hope too smile.gif

This post has been edited by guruboolez: Oct 13 2003, 16:15
Go to the top of the page
+Quote Post
JohnV
post Oct 13 2003, 17:27
Post #33





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



QUOTE (rjamorim @ Oct 13 2003, 05:59 PM)
Also, keep in mind that if he used q75, he would be flooded with whiner posts shouting "APPLES AND ORANGES! APPLES AND ORANGES!"

I don't see anybody shouting, even though MPC -radio gives clearly higher bitrate. I see just tweaking of some codecs, but not others. A bit mixed bag in my opinion.

Also, if the bitrate is tweaked on some codecs according to "classical baroc music 128kbps", I'd like to see it based on clearly more samples than 18. Let's say at least 5 or 10 CDs.. Why? Because the volume of the recordings can affect on the codec's bit allocation, regardless of music.

QUOTE
mp3 abr 134 : 7 615 221
QT AAC cbr 128 : 7 856 071
WMA 128 : 7 673 606
Nero : 7 537 148
Maybe under 128 kbps, but really close to others encoding.

Ok, hopefully I'm not counting wrong. Assuming QT AAC cbr 128 is 128kbps:
mp3 abr 134: 124.0 kbps
WMA pro 128 vbr-2pass 128: 125.0 kbps
Nero: 122.8 kbps
Vorbis -q4.25: 131.6 kbps
MPC -radio: 139.3 kbps
FAAC: 116.1 kbps
WMA9 pro q90 size is not visible anywhere, so can't really say...


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
rjamorim
post Oct 13 2003, 20:36
Post #34


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (JohnV @ Oct 13 2003, 01:27 PM)
I don't see anybody shouting, even though MPC -radio gives clearly higher bitrate. I see just tweaking of some codecs, but not others. A bit mixed bag in my opinion.

Erm.. read my post again. IF he did so, people would be shouting.

MPC radio indeed uses a higher bitrate, but not by a margin of 40kbps like would happen with WMA q75 vs. WMA q90.

Again, I completely agree with what Guruboolez did. For the "tweaking of some codecs, and not others": This is an ABR/VBR test, so allowing some codecs to scale for a little more or a little less is perfectly acceptable. But allowing a codec to scale to 40 kbps less is unacceptable! That would render completely unusable results for that codec.

I would say, from the fair amount of experience I have, that allowing a codec to scale to 10% more or less than the base bitrate (128, in this case) is acceptable. That's why I tried to limit the bitrates in my 64kbps test to scale in the range of 60 to 70kbps (WMA behaved badly, I admit). If you get higher deviation than 10%, issues of fairness and usefulness of the test start becoming critical.

Regards;

Roberto.


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
JohnV
post Oct 14 2003, 05:58
Post #35





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



QUOTE (rjamorim @ Oct 13 2003, 10:36 PM)
MPC radio indeed uses a higher bitrate, but not by a margin of 40kbps like would happen with WMA q75 vs. WMA q90.

Again, I completely agree with what Guruboolez did. For the "tweaking of some codecs, and not others": This is an ABR/VBR test, so allowing some codecs to scale for a little more or a little less is perfectly acceptable. But allowing a codec to scale to 40 kbps less is unacceptable! That would render completely unusable results for that codec.

Allowing codecs to scale 10% is acceptable, but imo you shouldn't mix 2 kind of testing methods in 1 test. Either you go with "quality level principle", or "bitrate tweaked principle", but both used in the same test for different VBR codecs (or some vague mixed method) and then refer to "10% acceptable scalability", I really don't know...

Also, I don't know if wma q90 (or q75) should be included in this test, because they just don't seem to give bitrates even close to 128 with Guru's samples. I tested the samples Guru posted with q90 and the bitrates were close to 150kbps and 170kbps. This is more than 10%. Some sample which had a long quiet section was 122 though.
So, you have this q90 on the upper part and FAAC with its 116kbps on the lower part. Doesn't really matter if FAAC was any other encoder, it still wouldn't be hard to say that it loses here, and I don't know what new information this brings. Those codecs which are "allowed" manually to scale, win.

I do think that the original test which didn't include wma9 pro q90 is more acceptable though, although I still don't like the mixing of testing principles.

This post has been edited by JohnV: Oct 14 2003, 07:56


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
askoff
post Oct 14 2003, 20:05
Post #36





Group: Members
Posts: 445
Joined: 23-December 02
Member No.: 4214



I think JohnV is at least 99% right. This same thing bothered me with rjamorim 128kbps extension test. Because most of the encoders arent quality based they should't adjust to one quality. And encoder witch have quality scale, they are not comparable with each others. I always though that we where trying to find most efficien encoder at close as possible to one bitrate.
So one comment to this thread and no more whining for me.
Go to the top of the page
+Quote Post
Soren
post Oct 14 2003, 22:47
Post #37





Group: Members
Posts: 141
Joined: 10-May 02
From: Quebec
Member No.: 2015



While we are in WMA topic, how can we encode wma pro in a easy method ? Can we use Windows media player, or the only options for wma pro encoding are dbpoweramp and the windows media encoder who give me really hard time to encode something with ?
Go to the top of the page
+Quote Post
bidz
post Oct 15 2003, 03:28
Post #38





Group: Members
Posts: 351
Joined: 27-December 02
From: Norway
Member No.: 4258



QUOTE (Soren @ Oct 14 2003, 01:47 PM)
While we are in WMA topic, how can we encode wma pro in a easy method ?  Can we use Windows media player, or the only options for wma pro encoding are dbpoweramp and the windows media encoder who give me really hard time to encode something with ?

i'd also like to know this. i also tried to encode a .wav with WMEncoder, and choose Q90 (VBR).. it made a 1-pass .wma that was identical in bitrate and size (down to bits) as to the same .wav file encoded in WMP9 with WMA9 Q90 (also that a 1-pass, cant seem to find a 2-pass option in any of these apps).. so that leads me to think that what i encoded with WMEncoder indeed was not a WMA9 Pro file..


--------------------
myspace.com/borgei - last.fm/user/borgei
Go to the top of the page
+Quote Post
guruboolez
post Oct 17 2003, 22:44
Post #39





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



I see two different kind of controversy here.
• First, "apple and oranges": I tweaked some encoders, and not another - that's bad.
• Second one: the choice of a wma VBR preset that isn't accurate.

I will try to answer to both problems with my little and approximate English. Sorry for all mistakes and my childish expression, but I really had to defend myself. Then, in a second message, I will add some capital precision about the version of Nero AAC I used, follow by another test. People interested by classical music may directly go to this last one.


I. MY CHOICES

I had to explain the methodology I used.
My goal was to build a good idea about encoders performance with the music I mostly listen. I had to find the best encoder for me, at a bitrate close to 130 kbps. Why 130 kbps and not less, or more? Not less, because the preliminary tests -very limited- I did (HE/LC-AAC, Vorbis) were not conclusive: I need something near transparency, and these low-bitrate encodings were too far from this state. Not more, because I'm short in space on my notebook. Therefore, something between 120 and 140 kbps is a good compromise. Of course, I prefer 120 to 140; but if at 140 kbps the quality jump is consequent, I can accept this slight sacrifice in disk space.

Then, for each encoder, I had to find the ideal setting: something between 120 and 140 kbps that will give me the best perceptible quality. I refused CBR, for obvious reasons : I ALWAYS read on HA than VBR is the best encoding method, followed by ABR, then by CBR., except some rare case (lame and low-mid bitrate VBR for example). I'm honestly surprised to read from JohnV that "with this kind of music CBR/ABR mode has the advantage, because VBR quality and bit allocation depends heavily on whether tonality estimation is working quite flawlessly or not" [JohnV, Posted: Oct 11 2003, 07:49 PM]. There are some people here listening to classical, and they would be happy to hear that ABR/CBR are better choice than VBR. Or is this affirmation applicable to low-mid bitrate only? Does it only mean that VBR is just using too much bits for a given quality and is, in one word, inefficient?
So, I chose VBR everywhere, except for QuickTime (no VBR, no ABR=>CBR only), LAME (no VBR preset=>ABR) and WMA (abr, for reasons I explained above, on a previous message).

After the choice of encoding method, I had to fix an adequate setting. I must precise that I didn't select the 18 samples before the listening phase, but progressively found them. In other words, I began with one sample, encoded it, listened to them, rated them, then I took another sample, encoded it, etc... This implies that I had fixed an encoding value for each encoder BEFORE I began the test. This implies too that I didn't proceed by any bitrate measurement phase. I fixed at the very beginning some settings, based on my own experience, others experience, or my total ignorance. Here is the explanation of my choice:

• Lame was set at ABR 134, because I have encoded a lot of classical with ABR for my hardware device, and I always saw lower bitrate (5-7 kbps) that what I asked. I use this general correction for every encoding I made or will make with LAME. Lame ABR is not accurate with classical : it's the case since Dibrom removed something like a bitrate correction during the --dm-preset era (he explicitly did that two years ago - I suppose that he could confirm this minor change, and if not, consider that claim as a memory flaw, or as a product of my fertile imagination). Therefore, --preset abr (x + 5...7 kbps) is what I would suggest for each people encoding classical. Note that final result was really nice: exactly the same size as QuickTime CBR 128 !

• Vorbis was set at -b 4.25, just because this value seemed to be the best according to the public results Roberto asked for some months ago. I used it too, because I often saw Vorbis giving lower bitrate with classical than with other genres at the same bitrate. Results told me that -b4 may be a better choice, but it was too late.

• Faac was set a q 128, because I totally ignore the behaviour of this encoder. I chose 128 value, just because Lame, QuickTime and WMA were intended to be close to 128. BTW, I include Faac encoder just because I had a free place in ABC/HR: I didn't expect Faac to be competitive – I just found interesting to have some idea about the quality of this encoder (used on some audioripper, as dBpowerAmp [Edit: after verification, dBPoweAmp is using fastenc AAC] or EasyCDDA Extractor). Same thing for TNS and M/S stereo: I didn't know how they will perform; I used just them: by ignorance of this encoder.

• WMA9 was set at ABR 128, just because I know pure VBR to be too much fluctuant (between 60 and 240 kbps for two different stereo classical tracks). ABR was the safest choice, even if it may not be the best one (VBR generally > ABR). I wrongly thought it will be the less problematic setting...

• I chose VBR for Nero, because some test I did some months ago were clearly in favor of -streaming against CBR 128. I haven't idea about ABR performances. The fact was, that I forgot that Nero AAC has an ABR mode - and even if I had remembered it, I would chose again VBR, for the theoretical benefits of this encoding method. I took -streaming profile, for two reasons. First, I noticed that average bitrate with pop music is visibly inferior to 128 kbps; but I noticed too that with classical (especially baroque, strings), bitrate is really close to 128 kbps, and sometimes higher than this symbolic value. Second reason: -normal profile is clearly too high for the bitrate range I set. Again, the fact gave me reason: with 2.5.5.8 encoder, average bitrate for the 18 samples was exactly 128 kbps - the encoding of the entire 18 discs would probably not being too far from this value.

• Same thing for PsyTEL : VBR against CBR/ABR, even if bitrate is commonly a bit higher than 128 kbps.

As you can see, I didn't perform any kind of tweak. The last proof of my claim is the mpc choice I made. I chose --radio setting, even if I knew that this profile gives higher bitrate that average value calculated on a wide variety of musical genres. I did this in order for two reasons:
• TO NOT TWEAK the encoder.
• TO KEEP the entire COHERENCY of my test (stay with the most immediate and easy to get setting for mpc : -q4, and not something like -q 3.68).
Just imagine the reproach, if I had compare mpc --thumb profile area, against the others competitor...

Obviously, I didn’t tweak anything for this test. The changes I made were made before the test: I used the settings I will use for all encodings I will make for listening purpose. I can’t be blame for having some knowledge about bitrate behaviour with classical listening. Is it a fault to anticipate some variations, and correct them with my own experience I accumulate during two years ? In my opinion, definitely not.
It's really hard for me to understand what you said by a "mixing of testing method". It's not very pleasant too to learn that the effort I did for this test is ruined because I "mix 2 kind of testing method", and that my results are just something like tolerable, "acceptable" [JohnV, Oct 14 2003, 05:58 AM].


II. WMA Q75 or WMA Q90


As I said before, I had in mind to test early music only, with a great dominant of baroque samples. The performances in quality, or the behaviour in bitrate variations, that both occurs on Metallica-like music, is something I don't care AT ALL. I’m not going to use an audio format giving excellent results on electronic but weak ones with violin, if I’m Paganini’s fan. In the same way, a person interested by a ~180 kbps audio format is not going to reject mpc –standard because complete works of Domenico Scarlatti (33 CDs) reach 220 kbps on average with the same preset.
Personally, I'm not rejecting pure knowledge, and I also would be interested by both bitrate and quality performance with Ride the lightning. But I'm not going to take these performances in count, on music I don't listen to, just in order to make pleasure to all people listening metal music every day.
After all, ff I never heard about HA, I would probably never see full album reaching 170 kbps with WMA9 PRO Q90. For my own purpose, based on my own -preset and future- experience of classical encoding, Q90 is the VBR setting close to 130 kbps.

Then, WMA VBR encoding performances are designed by a very abstract way: Q10, Q25...Q98. I don't see anywhere Microsoft claiming that Q75 is intended to be close to 120 kbps, nor that Q90 setting should be close to 180 kbps. Therefore, I found difficult to consider as a "codec problem" [JohnV,Oct 13 2003, 01:53 PM] the average bitrate of Q90 setting with classical music. I don't see why we should say that with classical music bitrate is "too low" [JohnV,Oct 13 2003, 01:53 PM]. Isn't the contrary valid too? Why isn't the bitrate considered as "too high" with others genre?
So, I can't judge the measured bitrate with classical as a problem. Of course, it would be one, if overall quality was bad, betraying any flaw of the psy-model or something similar. But it isn't the case. WMA VBR gives me something like 140 kbps (+10%) with classical music; therefore, I expect some improvement in quality compared to ABR 128. Quality is slightly improved compared to ABR (except increase in noise), therefore, VBR is doing its job.
As a conclusion, I would say that I didn't consider at all as a "problem", or as a fault, WMA Q90 performances: quality and bitrate are coherent, at least, on the 18 samples I tested. I need valid argument to accept the idea of a "problem": nobody on HA is blaming mpc --standard for the very low bitrate measured on piano encoding, or saying that's a "problem". Au contraire, everybody applaud on the clever behaviour of well-tuned VBR encoder. As long as no audible or unexpected problems are detected, a VBR encoder is allowed to reduce the needed bitrate without encountering any blame. This is at least what I conclude after reading HA during two years. The same policy used for mp3, mpc and vorbis should be applied for WMA PRO without an hesitation or hair-splitting.

I finally opted for Q90, just because I knew that it's close to 128 kbps. In order to help Roberto for his listening test, I've encoded two anthologies of classical music in order to build a good idea about bitrate behaviour of each profile. Two discs, but 30 tracks, from 30 different discs, including recording of various genre, period and even recording techniques (two or three of them were mono - some others very noisy). Average bitrate for these 30 tracks at Q90 is 131 kbps. At Q75, bitrate is 88 kbps. With the 18 samples I choose, coming from 18 different discs (all stereo), bitrate were very similar: 139 & 93 kbps. Incidentally, results were published, here, on HA. I even gave the links on page 1. Therefore, I was surprised to read the ask for encoding "5 or 10 CDs. Why? Because the volume of the recordings can affect on the codec's bit allocation, regardless of the music" [JohnV, Oct 13 2003, 05:27 PM]. 48 tracks & samples, coming from 48 different classical discs, are not enough for giving a positive idea about average bitrate?
Few others of your arguments are annoying me a bit – probably bad luck: "I tested the samples Guru posted with q90 and the bitrate were close to 150 kbps and 170 kbps". Three only are superior to 170 kbps (Dorilla, Questo core and Viva Rey Ferrando). 13 are inferior to 150 kbps. I know that I didn’t have uploaded all 18 files, so I can’t put the shame on you. Nevertheless, it’s a VBR test: and it surely not very conclusive to anticipate the average bitrate of the whole gallery with only the quarter of them. Using this biased argument against the validity of my test is something I don’t really understand. Nor like.
Q90 is, with the limited experience I have, and the limit I fixed for my test, the fairest challenger for a comparison with ABR or CBR 128 kbps. Q75 is clearly inferior to 100 kbps. Now, do you really consider as serious, or fair, or scientific, a test comparing WMA at 90 kbps against others encoders, measured between 120 and 140 kbps? Personally, I don't. To be honest, I found it unreasonable.
Other thing I don’t (literally – my limited English is guilty) understand: "those codecs which are 'allowed' manually to scale, win.". Does it mean that scalable encoders are real winners? Winners of the test are the two less tweakable encoders: QuickTime AAC (112-128-160 kbps steps) and WMAPRO ABR (128-192-256 kbps). And the biggest values aren’t the best one: vorbis (135 kbps) and mpc (142 kbps) were clearly behind the two strict 128 kbps encodings.



You don't like the idea of mixing different principles, I don't like it too.
Problem is, if people had to choose what they expect to be the best encoding at a given bitrate, they have the right to compare CBR, ABR and VBR, at a given bitrate. You can't forbid someone using a VBR preset, useful with the kind of music he listen to, just because this preset is useless with other kind of music. You can't forbid this person to compare this VBR setting with some CBR settings, because few encoders don't offer VBR or ABR method. Encoders are not developed for being played in ABC/HR only. Encoders are not developed for people listening various genre of music only. You are maybe right by saying that we need to separate two different groups of encoders for testing purpose, in order to respect some testing principle uniformity. But a castrated test is absolutely useless for practical purpose. If one people is asking for building a 200 kbps mp3 library, and hesitate between CBR 192 and --preset standard, what would be your answer? VBR preset? Or will this answer look something like: "I can't answer this, because the two settings are not directly comparable, and I don't like the idea of mixing two different testing principles"?

I did a test with the settings I would use for a practical and daily use. Nothing was arranged by me in order to scale the average bitrate of 18 grain-of-sand samples on a desired value. I used unilateral principle, simulated a real usage by using the same setting for both daily listening and occasional testing. In one word, it was totally coherent.

There's one moment where you're obliged to test each encoders, with their best weapons, even if the test looks like a confuse bric-ŕ-brac of encoding principles. That's what I made. I find it coherent, and useful in the same way.


Complete table: bitrate, size and notation

Warning: I obtained all bitrate values by using a simple rule of three, and using the average value of QuickTime AAC CBR 128 and WMA ABR 128 as reference. I removed all tags before calculation (except replaygain tags)


EDIT: Crossed dBpowerAmp name.

This post has been edited by guruboolez: Oct 18 2003, 15:51
Go to the top of the page
+Quote Post
guruboolez
post Oct 17 2003, 22:44
Post #40





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



The new .dll of Ivan Dimkovic I tested this week gave me the same bitrate than the encoder I used for the initial test. Audio stream is bit-to-bit identical. Explanation: I mixed up Nero5 and Nero6 encoder path. This is annoying, because it implies that I didn’t test the latest Nero encoder [2.5.5.8] during first listening test (8-challengers). The encoder used was an old version, from this summer, one of the first including he-aac profile. Exact number is 2.5.1.6…
I’m confused, especially for Ivan. I’ve just denied four month of his work. These months are probably more important for Nero AAC than for other encoders, as musepack or vorbis. Nero AAC is actually a young encoder, and progress is probably constant, and fast.

Therefore, before I start a new listening test, comparing old and experimental encoders, I decide to add a third encoder in the arena:
• 2.5.1.6 – old & uninteresting version of aacenc32.dll I tested first with 7 other challengers
• 2.5.5.8 – the latest released version of aacenc32.dll, bundled with Nero 6.0.19
• 2.5.6.2 – the experimental version of aacenc32.dll, named 2.5.6.2, that Ivan kindly sent me.
By testing 2.5.1.6 and 2.5.5.8, I can obtain a precise idea of the progress of the encoder, in order to evaluate 2.5.5.8 progress and to estimate the virtual position of this encoder in the 8-challengers test.
I initially planed to ask Ivan agreement before publishing results of an unofficial encoder. But results are so positive, and because I did a first and deplorable mistake last week, I have decided to bypass its authorization. Results are here:

http://membres.lycos.fr/guruboolez/AUDIO/t...Grille_Nero.htm

Note that notation should not be directly compared with notes I gave few days ago, with others competitors.



Initial comment:
- bitrate of 1.6 & .5.8 encoders are really close. 126 kbps against 128 kbps. More problematic for my purpose, the value I obtained for 2.5.6.2 –streaming: 148-149 kbps. +16%. Of course, it’s a less interesting bitrate for my purpose, and a really annoying one for my test (excessive if compared to most challengers). It’s the biggest deviation of the whole test (I’m not considering Q75 suggestion as a serious one: -27%). I honestly don’t have any idea of the average bitrate I would obtain with other musical genres. Nevertheless, for my favorite music, and the targeted bitrate area I’m looking for, this new Nero AAC encoder need a downgrade to –internet profile to stay in bitrate competiton.
Quality jump is really impressive, and bitrate inflation couldn’t be invoked to explain the whole progress. Some of the worst distortions audible with 2.5.5.8 encoder at –streaming profile (average 128 kbps) are indeed gone with 2.5.6.2 –internet (124 kbps).

Some progress are clearly audible between 2.5.1.6 and 2.5.5.8 release. For exemple, Dorilla sample: destroyed with the old encoder, now noise-free with the newest (though bad distortions). Passacaglia (organ sample) encoded with 2.5.1.6 was clearly the worse one during the 8-challengers test. With 2.5.5.8, it’s near transparency, as other formats. But if I except these two samples, the overall progress is really slim: small improvements on some samples, small degradation on others. Finally, the progress between 2.5.1.6 and 2.5.5.8 is real, but it’s more like a bug correction, with audible and considerable effect on limited situations. 2.5.5.8 is probably closer to PsyTEL performances than to Faac ones. That’s a good step, but the major one, for my needs, will be the next aacenc32.dll release.



Bitrate and size are available on the same table as before:
membres.lycos.fr/guruboolez/AUDIO/test_MF_early/Bitrate_Table_complete.htm


Note that I carefully checked the integrity of each PCM and each encoded files. I noticed some corruption and discordance between some encoding sets, and I corrected them.
Go to the top of the page
+Quote Post
JohnV
post Oct 17 2003, 22:58
Post #41





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



QUOTE (guruboolez @ Oct 18 2003, 12:44 AM)
The new .dll of Ivan Dimkovic I tested this week gave me the same bitrate than the encoder I used for the initial test. Audio stream is bit-to-bit identical. Explanation: I mixed up Nero5 and Nero6 encoder path. This is annoying, because it implies that I didn’t test the latest Nero encoder [2.5.5.8] during first listening test (8-challengers). The encoder used was an old version, from this summer, one of the first including he-aac profile. Exact number is 2.5.1.6…

Ok, well it's very good you found this rather umm fundamental issue..
QUOTE
More problematic for my purpose, the value I obtained for 2.5.6.2 –streaming: 148-149 kbps. +16%. Of course, it’s a less interesting bitrate for my purpose, and a really annoying one for my test (excessive if compared to most challengers).
I don't think you should publicly post results about development builds. Send your results to Ivan privately. First of all, like you noticed, the profiles of 2.5.6.2 and further are not yet tweaked to correspond the changes in psychoacoustics.


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
guruboolez
post Oct 17 2003, 23:06
Post #42





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (JohnV @ Oct 17 2003, 10:58 PM)
I don't think you should publicly post results about development builds. Send your results to Ivan privately. First of all, like you noticed, the profiles of 2.5.6.2 and further are not yet tweaked to correspond the changes in psychoacoustics.

As I said it, I first had in mind to do it. But I did a big mistake last week, giving to a wrong and outdated version of Nero AAC encoder, a really bad notation. Therfore, I think that these new promising results would be welcome. There are some fan of Ivan works: I had to compensate for my mistake...
There are only results, comparing Nero to Nero. Exciting values, or promising words as yours in first page, are so different?
Nevertheless, if Ivan want it, I can remove the link and correct the html table (or you, JohnV, if Ivan is asking this before I read it).

This post has been edited by guruboolez: Oct 17 2003, 23:07
Go to the top of the page
+Quote Post
JohnV
post Oct 17 2003, 23:13
Post #43





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



QUOTE (guruboolez @ Oct 18 2003, 01:06 AM)
QUOTE (JohnV @ Oct 17 2003, 10:58 PM)
I don't think you should publicly post results about development builds. Send your results to Ivan privately. First of all, like you noticed, the profiles of 2.5.6.2 and further are not yet tweaked to correspond the changes in psychoacoustics.

As I said it, I first had in mind to do it. But I did a big mistake last week, giving to a wrong and outdated version of Nero AAC encoder, a really bad notation. Therfore, I think that these new promising results would be welcome. There are some fan of Ivan works: I had to compensate for my mistake...
There are only results, comparing Nero to Nero. Exciting values, or promising words as yours in first page, are so different?
Nevertheless, if Ivan want it, I can remove the link and correct the html table (or you, JohnV, if Ivan is asking this before I read it).

Well, it's fine by me, as long as people remember that your 2.5.6.2 results are results from a development build which is still evolving before the next stable..


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
guruboolez
post Oct 17 2003, 23:21
Post #44





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE
Well, it's fine by me, as long as people remember that your 2.5.6.2 results are results from a development build which is still evolving before the next stable..

Should be OK now wink.gif

This post has been edited by guruboolez: Oct 17 2003, 23:23
Go to the top of the page
+Quote Post
JohnV
post Oct 18 2003, 03:12
Post #45





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



Thanks Guru.

What comes to your explanation about the test.. well, it was long. smile.gif
The biggest questional thing still imo was the q90 (the winner), which has "offset" very high, and gives according to you about 140 with these samples. As I said earlier, imo neither q75 or q90 was very good for this test. Imo q90 is fundamentally in different class than others. It's like driving a Ferrari 140 km/h and claiming that it's in the same class with an old Volkswagen which can barely reach 120km/h, because they are going nearly as fast. Considering the q90 overall average bitrate is very high (Ferrari), and vbr bitrate with your samples near 140, and considering that between 120-140 the quality difference of vbr codecs can be relatively much higher than for example between 140-160...

Anyway, this is just my opinion and others can and probably will disagree.
I don't think it does anybody any good to continue chewing this. In any case, your test wasn't flawed in that sense, that there's reason to invalidate it. Imo it could have been a bit better in some ways, but I think that is also a matter of opinion. Well the usage of a quite old Nero codec was quite considerable technical mistake though, but fortunately you noticed it and reacted smile.gif (You might want to correct your original message's Nero codec version number also)

And in any case HA needs blind testings and testers with good hearing like you, so I hope you weren't discouraged (from the length and tone of your explanation, it seems you weren't, which is good. wink.gif )


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
guruboolez
post Oct 18 2003, 11:56
Post #46





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



Thank for your positive answer. I spent long hours to wrote my defence, and I expected your comments. I'm not to continue the debate, because I don't want to spend my week-end in an english dictionnary wink.gif

The first message is now corrected, including precisions about the encoder version I unfortunately used.

I'm not discouraged anymore (but I was last week). Afterall, I'm doing tests for my own usage, not only for the others.

Finally, I noticed that WMA9PRO VBR90 is probably flawed: I've saw some low-intensity stereo piano tracks with a bitrate inferior to 60 kbps, and here, distortions are easily noticeable (but I didn't ABX them). If I could confirm this, this will end the debate: ABR128 would be the safest competitor (and the more universal) for this bitrate area.
Go to the top of the page
+Quote Post
LagunaSol
post Oct 19 2003, 00:06
Post #47





Group: Members
Posts: 13
Joined: 21-May 03
Member No.: 6737



Quicktime 6.4 (released with iTunes for Windows) claims enhanced AAC encoding. By enhanced I don't know if Apple means speed or quality or both (can I hope?). How 'bout re-running your test using just WMA Pro and AAC using QuickTime 6.4? Enquiring minds want to know...
Go to the top of the page
+Quote Post
bidz
post Oct 19 2003, 03:46
Post #48





Group: Members
Posts: 351
Joined: 27-December 02
From: Norway
Member No.: 4258



Kind of off-topic, but still a bit on-topic (as i didnt want to start a single thread for a simple question) :

What would be the best way to use the WMA9 Pro codec in? as there is no 16-bit 44khz 2-channel preset to use, so my question is, is there a negative side on using 24-bit on samples that are 16 bit ?

I also encoded the From Dusk Till Dawn soundtrack with both WMA9 Pro (Q90, VBR, 44 kHz, 2 channel 24 bit) and WMA9 Std (Q90, VBR, 44 kHz, stereo 16 bit), and i noticed that the filesizes was very different, the filesize off the WMA9 Pro encode was actually alot smaller than the one encoded with WMA9 Std - examples:

Track 13, Tito & Tarantula - After Dark:
---------------------------------------------
WMA9 Pro filesize: 5,16 MB (5 415 279 bytes)
WMA9 Pro bitrate: 171 kbps

WMA9 Std filesize: 6,31 MB (6 625 061 bytes)
WMA9 Std bitrate: 224 kbps

Track 2, The Blasters - Dark Night:
----------------------------------------
WMA9 Pro filesize: 4,51 MB (4 735 743 bytes)
WMA9 Pro bitrate: 180 kbps

WMA9 Std filesize: 5,09 MB (5 345 409 bytes)
WMA9 Std bitrate: 205 kbps


To be honest, i think it's strange that a 24bit file is smaller than a 16bit one, and i would also asume that the Pro codec would generate bigger filesizes/bitrates at the same quality level ? Has anyone done a good test off the WMA9 Std. codec and compared it with the Pro codec, at Q90 (VBR, no 2-pass "ABR") ?


--------------------
myspace.com/borgei - last.fm/user/borgei
Go to the top of the page
+Quote Post
Yodule
post Oct 19 2003, 05:51
Post #49





Group: Members
Posts: 6
Joined: 28-May 03
Member No.: 6861



QUOTE (Soren @ Oct 14 2003, 05:47 PM)
While we are in WMA topic, how can we encode wma pro in a easy method ?  Can we use Windows media player, or the only options for wma pro encoding are dbpoweramp and the windows media encoder who give me really hard time to encode something with ?

There comes with wmencoder (or wmencoder sdk, don't remember) a windows script to encode a file (windows media encoding script).

Later, I found on usenet an modified version where you can specify the artist/genre/album/etc tags.

With that script I can now encode WMA9Pro or Std directly from cdex. I can mail a copy of the script if somebody is interessted.

I Just made my own little experiment with WMa9pro 128K vbr 2 pass with the first track of AC/DC album "High voltage". The intro of that track (just a raw guitar) seems to be a killer for nero AAC @ 128 kbits (sounds like a cellular phone), for wma9std 128 vbr 2 pass (sounds like a robot) and for wma9pro 128 vbr 2 pass (you hear cracks, just like a vinyl LP !).

My solution was to encode wma9 Q90. Both Wma9pro or std seems to sound good, but the wma9std file was bigger (I remember it was >5Megs) than the wma9pro one (it is 4.39 Megs). Fast generalisation conclusion : wm9pro q90 seems good quality/compression ratio and safer than wma vbr 2 pass.
Go to the top of the page
+Quote Post
bidz
post Oct 19 2003, 06:22
Post #50





Group: Members
Posts: 351
Joined: 27-December 02
From: Norway
Member No.: 4258



QUOTE (Yodule @ Oct 18 2003, 08:51 PM)
With that script I can now encode WMA9Pro or Std directly from cdex. I can mail a copy of the script if somebody is interessted.


Please send it to me by email: borge_i@hotmail.com

QUOTE
My solution was to encode wma9 Q90. Both Wma9pro or std seems to sound good, but the wma9std file was bigger (I remember it was >5Megs) than the wma9pro one (it is 4.39 Megs). Fast generalisation conclusion : wm9pro q90 seems good quality/compression ratio and safer than wma vbr 2 pass.


WMA9 Pro/Std 2-pass isn't actually VBR, its more like ABR. 1-pass is true VBR (atleast it seems this way).

As in my examples a bit up, WMA9 Std filesize is much bigger than WMA9 Pro.. but still people say that WMA9 Pro has much better encoding quality..

dBPowerAmp Music Converter is very good for ripping/encoding to WMA9 Pro i think, but i'd still like to try your script, maybe i could use that with EAC then.

This post has been edited by bidz: Oct 19 2003, 06:23


--------------------
myspace.com/borgei - last.fm/user/borgei
Go to the top of the page
+Quote Post

3 Pages V  < 1 2 3 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 23rd August 2014 - 07:19