IPB

Welcome Guest ( Log In | Register )

14 Pages V  « < 3 4 5 6 7 > »   
Reply to this topicStart new topic
Autumn 2006 Listening Test, MP3 @ 128 kbps vs. Multiformat @ 80 kbps
Sebastian Mares
post Aug 15 2006, 23:53
Post #101





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



So, for the encoder decision and settings, do you think some pre-tests are appropriate? Obviously, not all encoders can be included to have a comparison between iTunes CBR and iTunes VBR, FhG CBR and FhG VBR, etc. What I know is that iTunes VBR performed very bad in Roberto's test. That's why I thought CBR should be used now. As for FhG, I always thought their CBR engine would also be better than their VBR one, but as you guys said, this has to be checked. However, this has to be checked in some pre-tests because there are already enough contenders.

I am hitting the sack now. smile.gif

This post has been edited by Sebastian Mares: Aug 15 2006, 23:54


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
guruboolez
post Aug 15 2006, 23:54
Post #102





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (Sebastian Mares @ Aug 16 2006, 00:32) *
iTunes' VBR mode is worse than its CBR mode.

Did I miss the listening test(s) proving that? Do you have a link?
The problem with iTunes (but correct me if I'm wrong) MP3 VBR is the following one:
1/ iTunes doesn't offer a scalable VBR engine (like HELIX or Fastenc does); you can only use "VBR" as option for fixed bitrate settings (like 96 kbps VBR, 112 kbps VBR, 128 kbps VBR etc... with nothing in the middle). It's a bit like current LAME VBR scale.
2/ But iTunes also uses the "target" bitrate as the minimal one per frame. In clear, if you select "192 kbps VBR", the bitrate will vary from... 192 kbps up to 320 kbps. "Low complexity" samples won't compensate for the bloating bitrate of "complex" samples. As a consequence the average bitrate of this setting could only be higher than 192 kbps.

In our case, "128 kbps + VBR" iTunes setting will for sure correspond to an average bitrate superior to 128 kbps. How greater? I can't say. But it will be superior.
An alternative choice would be "112 kbps + VBR" which would correspond to a bitrate higher than 112 kbps and --who knows-- maybe as close to 128 kbps than the previous setting. But as Roberto explained it this choice would handicap iTunes encoder (as it handicap it in 2004).
So assuming that iTunes 112@VBR is lower quality than 128@CBR, we still don't know how iTunes 128@VBR would perform against CBR@128. As I said, VBR@128 won't go below 128 kbps. Therefore it's hard to imaging this setting performing poorer than 128@CBR! So what we really need to know is the average bitrate of iTunes VBR at "128 kbps". If the average bitrate isn't too far from 128 kbps then iTunes VBR would easily be include in the test. If the bitrate is a bit too high (138...140 kbps) then we could easily increase a bit the setting for HELIX, FHG (and also GOGO). Then we would perform a listening test at ~135...140 kbps instead of a ~128...132 kbps one. Not a big issue in my opinion.


QUOTE
As for FhG, MMJB could be an option if VBR is needed. Aren't both WMP's and MMJB's MP3 encoders based on FastEnc?
Adobe Audition also uses Fhg encoder with a nice VBR scale.

This post has been edited by guruboolez: Aug 15 2006, 23:55
Go to the top of the page
+Quote Post
Sebastian Mares
post Aug 16 2006, 00:01
Post #103





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



I find it hard to believe that 6 kbps difference between iTunes and LAME can make such a difference in quality. And if iTunes VBR implementation is so good, why didn't it allocate more bits for hard parts like LAME did?

Anyways, I will let iTunes encode my music collection (electronic, rock, instrumental, pop, could add some classical music too) and see what it comes up with.

This post has been edited by Sebastian Mares: Aug 16 2006, 00:02


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
greynol
post Aug 16 2006, 00:03
Post #104





Group: Super Moderator
Posts: 10257
Joined: 1-April 04
From: San Francisco
Member No.: 13167



QUOTE (Sebastian Mares @ Aug 15 2006, 15:38) *
I think we are nitpicking regarding the title. I used the term "@ 128 kbps" because of the tradition. I can also call it "@ ca. 43 * Pi kbps".

Again, I happen to agree with guruboolez. I don't believe we're picking nits. I think that 128 kbps implies CBR and is therefore misleading. 130, OTOH, is clearly VBR. IMO, tradition is usually never a good reason to do anything and I have taken issue with this idea of passing apples off as oranges with previous tests but I don't believe I've said anything about it until now. I apologize for voicing my opion too strongly.

As far as genres and killer samples go, bitrates can vary. I don't think you'd want to change the quality setting just to achieve a certain bitrate because of a certain genre. This also reinforces the point of not putting CBR up against VBR. Perhaps using samples of metal can demonstrate this, especially where the use of VBR will result in bitrate increases of greater than 10%?


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
Sebastian Mares
post Aug 16 2006, 00:12
Post #105





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Man, didn't I say I wanted to hit the sack. Now it's 1 AM again and I am still here.

OK, I will change the title - no problem.


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
guruboolez
post Aug 16 2006, 00:15
Post #106





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (Sebastian Mares @ Aug 16 2006, 00:53) *
So, for the encoder decision and settings, do you think some pre-tests are appropriate?

Pre-test would enlight us a lot if they're done properly (i.e. several samples, several people). A big pre-test performed by a single person whistling.gif has a well-known limitation, and a small-one performed by several people but on a small selection of samples may also lead to false conclusion.
And what we need here is more than one pre-test. Suggestions for pre-tests:
- iTunes CBR vs iTunes VBR
- Helix VBR vs Helix CBR
- Helix default setting vs Helix 'tuned' setting
- Fhg VBR vs Fhg CBR
- Fhg from {one software} vs Fhg from {another software}
- LAME -V5 vs LAME -V5 --vbr-new
- LAME 3.97 vs LAME 3.98

If someone start a pre-test to "extract" the best setting for one encoder (let's say iTunes as example) then it would be unfair to refuse the same treatment for all other competitors (excepted LAME maybe which is the one we're more familiar with). iTunes, Helix, Fhg... our lack of knowledge is the same for all these encoders and I wouldn't really see as fair to privilege one over the other just before the test begins. On the other side, I can't imagine how great the enthousiasm would be to the idea of performing three pre-tests as a prelude to another one dedicated to MP3 encoders. smile.gif

Not to mention than even if pre-tests are done some people would still attack your choice, especially if a fight against CBR vs VBR will occur :/

This post has been edited by guruboolez: Aug 16 2006, 00:48
Go to the top of the page
+Quote Post
Sebastian Mares
post Aug 16 2006, 00:20
Post #107





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Jesus, that's a lot of pre-tests. I am thinking of another possibility... Only thing that comes into my mind would be running two tests: one with all encoders using CBR and one with all encoders using VBR. However, this is rather complicated. We would have to use the same samples, but then the factor tester could have an impact on the results (different testers or even moods = different results).


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
guruboolez
post Aug 16 2006, 00:42
Post #108





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (Sebastian Mares @ Aug 16 2006, 01:01) *
I find it hard to believe that 6 kbps difference between iTunes and LAME can make such a difference in quality.

Don't forget that the bitrate table is based on short samples (selected ones, who all tend to use more bits than average). It implies that iTunes 112@VBR average bitrate would logically be lower and therefore that the difference with LAME ABR bitrate is therefore higher than 6 kbps.

QUOTE
And if iTunes VBR implementation is so good, why didn't it allocate more bits for hard parts like LAME did?
It's indeed a good question. Someone said once as hypothesis that the limited bitrate's variations of this encoder is linked to the (old) iPod CPU clock issue. I tend to believe that this limitation simply correspond to the will of keeping the bitrate as close to the mentioned one (128@VBR with iTunes is indeed close to 128 kbps whatever the situation and it's probably what most users are expecting from this setting).

N.B. what is interesting us at the moment is not to see if iTunes VBR is "so good" but rather if it isn't "poor" (in order to see if CBR would be more suitable than VBR). One advantage of iTunes trick (i.e. using the target bitrate as the minimal floor per frame) is at least to prevent the encoder of big encoding mistakes. I expect from iTunes CBR and VBR to be very close each others (quality-wise) simply because both modes are always leading to the same bitrate (approximately). With LAME on the contrary you can expect from VBR to sound much better on critical material (some samples could reach 200 kbps with -V5) but also to sound poorer in some cases (it was the case with quiet samples - problem partially fixed with --athaa-sensitivity 1 switch and now close to be totally fixed with 3.98 alpha 6).



QUOTE (Sebastian Mares @ Aug 16 2006, 01:20) *
Jesus, that's a lot of pre-tests. I am thinking of another possibility... Only thing that comes into my mind would be running two tests: one with all encoders using CBR and one with all encoders using VBR. However, this is rather complicated. We would have to use the same samples, but then the factor tester could have an impact on the results (different testers or even moods = different results).

I have something more easy: let people having choice of testing either CBR-only settings or VBR-only ones. A poll could be an answer.

If a majority of people are rather interested by 128 kbps CBR => test everything with CBR. Then no bitrate table fight. Only problem: people interested to optimize the quality for their DivX/Portable player... at moderate bitrate won't learn anything by a test focusing on... unoptimized settings (CBR instead of VBR).

If a majority of people are rather interested by 128 kbps VBR => test everything with VBR and discard all encoder which doesn't offer either VBR coding or simply a ~128 kbps VBR setting.

Simple, isn't it? laugh.gif

This post has been edited by guruboolez: Aug 16 2006, 00:51
Go to the top of the page
+Quote Post
HbG
post Aug 16 2006, 00:47
Post #109





Group: Members
Posts: 289
Joined: 12-May 03
From: The Hague
Member No.: 6555



Thanks for accepting l3enc. smile.gif Very long ago i had a few mp3's with a short burst of noise at the start, now i know why.

As for the other contenders, since we are testing encoders i believe that each encoder should contend at the most optimal settings for the target bitrate, and it doesn't matter if that's cbr or vbr, just whatever is best.

Indeed a pre-test may be a good way to determine the most optimal settings, but that really means we're doing the developers' homework.


--------------------
Veni Vidi Vorbis.
Go to the top of the page
+Quote Post
Dologan
post Aug 16 2006, 04:41
Post #110





Group: Members (Donating)
Posts: 478
Joined: 22-November 01
From: United Kingdom
Member No.: 519



Ahhh, am I too late?? I wanted to suggest slipping a standard iTunes AAC @ 128 kbps as "pseudo-high" anchor. Might be interesting to see if you could say AAC really is "out of MP3's league" or if MP3 in general (...ok, Lame tongue.gif) is a good contender. Ok, shut me up... blush.gif

(Hey! First post in many many months! Yay!)

This post has been edited by Dologan: Aug 16 2006, 04:41
Go to the top of the page
+Quote Post
Shade[ST]
post Aug 16 2006, 04:56
Post #111





Group: Members
Posts: 1189
Joined: 19-May 05
From: Montreal, Canada
Member No.: 22144



QUOTE (Dologan @ Aug 15 2006, 23:41) *
Ahhh, am I too late?? I wanted to suggest slipping a standard iTunes AAC @ 128 kbps as "pseudo-high" anchor. Might be interesting to see if you could say AAC really is "out of MP3's league" or if MP3 in general (...ok, Lame tongue.gif) is a good contender. Ok, shut me up... blush.gif

(Hey! First post in many many months! Yay!)

In the multiformat test, iTunes AAC was statistically tied with LAME -V5 --vbr-new. sorry tongue.gif
Go to the top of the page
+Quote Post
jmartis
post Aug 16 2006, 08:36
Post #112





Group: Members
Posts: 381
Joined: 9-April 06
From: Czech Republic
Member No.: 29311



As I mentioned above, the FHG Fastenc encoder forces simple stereo in VBR mode. So I suggest testing it in CBR (even if all other contenders are in VBR), because I'm sure it will perform better with CBR.
Go to the top of the page
+Quote Post
Sebastian Mares
post Aug 16 2006, 09:00
Post #113





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Guru, a poll would ease up everything, but in the end, we won't know if CBR or VBR was better for certain encoders. For this purpose, we really have the two options only: lots of pre-tests or two big listening tests.


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
HbG
post Aug 16 2006, 10:23
Post #114





Group: Members
Posts: 289
Joined: 12-May 03
From: The Hague
Member No.: 6555



If you view the test as a study that should have some authorative value over which encoder is best and how they compare, a poll isn't very scientific.

What about mailing the developers of each encoder and asking them?


--------------------
Veni Vidi Vorbis.
Go to the top of the page
+Quote Post
Sebastian Mares
post Aug 16 2006, 10:29
Post #115





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Well, for small groups or individual developers (LAME, AoTuV, etc.) this is an option, but who is responsible for iTunes' MP3 encoder? Who is responsible for WMPs encoder? And I already made the experience with writing mails to big companies and the time it takes to receive a reply.


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
Sebastian Mares
post Aug 16 2006, 10:53
Post #116





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Going to encode to iTunes VBR @ 128 kbps to see what the average bitrate is. Should the other settings be like in Roberto's test?


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
Egor
post Aug 16 2006, 11:06
Post #117





Group: Members
Posts: 826
Joined: 29-September 04
Member No.: 17374



QUOTE (Sebastian Mares @ Aug 16 2006, 16:29) *
Who is responsible for WMPs encoder?

I think they are: Audio & Multimedia. But what's not clear with their encoder?
Go to the top of the page
+Quote Post
Sebastian Mares
post Aug 16 2006, 11:21
Post #118





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



While the encoder is based on FastEnc, I am sure MS did some own tuning. And what is unclear? If we should use VBR or CBR.

This post has been edited by Sebastian Mares: Aug 16 2006, 11:21


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
fpi
post Aug 16 2006, 11:25
Post #119





Group: Members
Posts: 58
Joined: 24-October 05
Member No.: 25326



I don't understand the point in testing CBR when a VBR option is available. As far as I know the only advantage of VBR should be its superior quality at the same average bitrate of CBR. If, for a particular codec, VBR has a lower quality than CBR, then it's a developers fault, in that case they should have provided only the CBR mode. So I don't see the point in testing CBR when a VBR option is available and that VBR option is not tagged by the developers as "experimental" or "you should use CBR, VBR option is provided here only to be useful for that particular case....".
Go to the top of the page
+Quote Post
Sebastian Mares
post Aug 16 2006, 11:36
Post #120





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



WMP doesn't come with a VBR option in first place. Maybe MS tuned the CBR encoder only and then focused on WMA - no idea. The problem with FhG encoders - as far as I can tell - is that each company might have done their own tunings unless this was prohibited from FhG's side. Therefore, you cannot say that FhG performed like this or that.


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
guruboolez
post Aug 16 2006, 11:37
Post #121





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (jmartis @ Aug 16 2006, 09:36) *
As I mentioned above, the FHG Fastenc encoder forces simple stereo in VBR mode. So I suggest testing it in CBR (even if all other contenders are in VBR), because I'm sure it will perform better with CBR.

It's maybe for a good reason, who knows... The feature, or this bug, is present for years. Even Audition 2.0 (Trial version) embedd a MP3 encoder forcing joint-stereo. On the other side, didn't Fastenc joint-stereo suffer from another kind of bug (stereo collapse or something like that)?

QUOTE
we won't know if CBR or VBR was better for certain encoders.

QUOTE
If you view the test as a study that should have some authorative value over which encoder is best and how they compare, a poll isn't very scientific.

It doesn't have a scientific purpose. And it's not intended to tell us which kind of coding is supposed to be better for each encoder. It's just a way for us to make a decision and to avoid inextricable problems. And yes, final decision would appear as an arbitrary one smile.gif
If people are interested to see how various MP3 encoders perform compared to LAME, I'd tend to say that it wouldn't be a bad idea to test them all with similar settings: VBR against VBR, CBR against CBR. VBR is per default supposed to offer a better quality (or efficiency) than CBR, especially on hard-to-encode samples (precisely what we're used to include in listening tests). HELIX, Apple, Fraunhofer... have all put energy and money to develop a VBR mode; this mode is offered to the public; there is no contraindication nor warning against VBR in the manual telling to users that VBR isn't tuned enough. And as far as I can remember, we didn't mail to Apple developers neither the Microsoft ones to see if we had to prefer CBR over VBR with iTunes AAC or WMAPro (last 130 kbps multiformat listening test).

I'm not against the idea of questionning developers themselves, but as you said, answer may take a long time. I'm not against making a pre-test, but it looks like filling ourselves a hole in the encoder/software's manual (and I'm very sceptical about the success of such pre-tests -- which are requiring as energy as the final test: we can't count on a pre-test to help us and spoil it by a lack of samples/testers). The idea of having two distinct tests (one for CBR, another for VBR) is also a good one, but again I'm not sure that many people are interested to spend free time to test twice what is often considered as an outdated format (MP3), including what is often perceived as outdated encoders (Fhg, iTunes), with one test dedicated to what is considered with right as a wrong coding method (CBR).

... just my 2 ct smile.gif

This post has been edited by guruboolez: Aug 16 2006, 11:40
Go to the top of the page
+Quote Post
Sebastian Mares
post Aug 16 2006, 11:38
Post #122





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Isn't the stereo collapse bug something in the unofficial encoder(s) only?


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
jmartis
post Aug 16 2006, 11:43
Post #123





Group: Members
Posts: 381
Joined: 9-April 06
From: Czech Republic
Member No.: 29311



QUOTE (guruboolez @ Aug 16 2006, 12:37) *
... On the other side, didn't Fastenc joint-stereo suffer from another kind of bug (stereo collapse or something like that)?

This bug is just in the Fastencc from rarewares, it is not present in the newer (WMP) fastenc.
Go to the top of the page
+Quote Post
Sebastian Mares
post Aug 16 2006, 11:44
Post #124





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



QUOTE (guruboolez @ Aug 16 2006, 12:37) *
And as far as I can remember, we didn't mail to Apple developers neither the Microsoft ones to see if we had to prefer CBR over VBR with iTunes AAC or WMAPro (last 130 kbps multiformat listening test).


WMAPro is developed entirely by Microsoft and WMP / WME offer VBR encoding. WMP's MP3 encoder on the other hand is CBR only. There is no VBR option provided. Using MMJB or AA is an option, but just like for WMP and its CBR encoder, the encoders are not only FhG, but FhG + company tunings.


QUOTE (jmartis @ Aug 16 2006, 12:43) *
QUOTE (guruboolez @ Aug 16 2006, 12:37) *

... On the other side, didn't Fastenc joint-stereo suffer from another kind of bug (stereo collapse or something like that)?

This bug is just in the Fastencc from rarewares, it is not present in the newer (WMP) fastenc.


Yes, and the fastencc from RRW is not an official build as far as I know. Some developer published the libraries and some other developers wrote a front end for them (AFAIK)


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
Sebastian Mares
post Aug 16 2006, 12:01
Post #125





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



iTunes encoded 47 tracks already and none of them has a bitrate higher than 140 kbps. The encoding speed (WAV to MP3) on my Pentium 4, 3 GHz, Hyperthreading and 1 GB RAM is 20.8x.

For longer tracks (over 4 minutes), the speed reaches 24x.

This post has been edited by Sebastian Mares: Aug 16 2006, 12:08


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post

14 Pages V  « < 3 4 5 6 7 > » 
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 28th November 2014 - 12:14