IPB

Welcome Guest ( Log In | Register )

35 Pages V   1 2 3 > »   
Closed TopicStart new topic
Multiformat 128 kbps Listening Test, Pre-Test Discussion
Sebastian Mares
post Nov 12 2005, 19:56
Post #1





Group: Members
Posts: 3630
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Greetings!

I am planning to start a multiformat listening test at 128 kbps on November, 30th which should end on December, 11th. The test start can be postponed if necessary.

Edit: The test has been postponed: Stard date is December 5th and end date is December 25th if everything goes well.

Edit: The discussion about codecs is now over.

The following codecs are going to be included:
  • iTunes / QuickTime AAC
  • Nero Digital 4.2.1.0
  • LAME 3.97
  • Ogg Vorbis, AoTuV 4.5
  • WMA Professional 9.1
  • Shine as low anchor
I'm open to suggestions regarding the settings to use. The only thing that's for sure is iTunes / QT AAC which will be used at 128 kbps VBR.
I'd like to ask for Garf's or Ivan's opinion regarding Nero Digital (CBR or VBR, etc.).
As for the rest of the formats, suggestions from the community are welcome, too.

I have some samples in mind already and we'll start discussing them soon. While at it, would you guys prefer 18 or 12 samples?

Regards,
Sebastian

This post has been edited by Sebastian Mares: Nov 29 2005, 16:47


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
de Mon
post Nov 12 2005, 20:18
Post #2





Group: Members
Posts: 474
Joined: 1-December 02
Member No.: 3940



IMHO
1. MusePack - on the last test it performed very well at that bitrate.
2. WMA - lot of people use it - it would be good to prove WMA is not the best codec as lot of people in the world think.
So my vote for MusePack and WMA.

This post has been edited by de Mon: Nov 12 2005, 20:20


--------------------
Ogg Vorbis for music and speech [q-2.0 - q6.0]
FLAC for recordings to be edited
Speex for speech
Go to the top of the page
+Quote Post
Raptus
post Nov 12 2005, 20:28
Post #3





Group: Members
Posts: 73
Joined: 22-February 04
From: Germany
Member No.: 12191



I'm with de Mon.
Go to the top of the page
+Quote Post
kwanbis
post Nov 12 2005, 20:52
Post #4





Group: Developer (Donating)
Posts: 2362
Joined: 28-June 02
From: Argentina
Member No.: 2425



wma standard and pro


--------------------
MAREO: http://www.webearce.com.ar
Go to the top of the page
+Quote Post
kennedyb4
post Nov 12 2005, 20:57
Post #5





Group: Members
Posts: 772
Joined: 3-October 01
Member No.: 180



Hi. Thanks for your efforts in organizing the test.

I would suggest skipping musepack as well for the reasons already mentioned and also because it is rarely used at this bitrate.

WMA standard would be the low anchor here I think.If WMA Pro is finding hardware support these days it might be useful to have it blind tested as well.

It breaks my open source loving heart that FAAC is not more actively developed. If it has not changed significantly then testing is probably not warranted at this time.

$.02
Go to the top of the page
+Quote Post
rjamorim
post Nov 12 2005, 21:22
Post #6


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (Sebastian Mares @ Nov 12 2005, 04:56 PM)
There is room for two other competitors or one competitor and a low anchor. Maybe MusePack and some version of WMA (Professional or Standard)?


There should be one WMA at least, otherwise the retards populating Slashdot will claim it wasn't tested because we are afraid our beloved codecs would be owned badly by it.

I wouldn't feature MPC. But that's what you should expect from an MPC hater like me.

QUOTE
While at it, would you guys prefer 18 or 12 samples?
*


18. Otherwise a well know person around here will use the few samples as an excuse for his format of choice not winning, in case it loses (and if it wins, he'll stay quiet)...

This post has been edited by rjamorim: Nov 12 2005, 21:26


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
Sebastian Mares
post Nov 12 2005, 22:10
Post #7





Group: Members
Posts: 3630
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



QUOTE (kennedyb4 @ Nov 12 2005, 09:57 PM)
WMA standard would be the low anchor here I think.
*


For the low anchor, I suggest using something that sounds bad for sure, like l3enc.


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
kennedyb4
post Nov 12 2005, 22:26
Post #8





Group: Members
Posts: 772
Joined: 3-October 01
Member No.: 180



QUOTE (Sebastian Mares @ Nov 12 2005, 04:10 PM)
QUOTE (kennedyb4 @ Nov 12 2005, 09:57 PM)
WMA standard would be the low anchor here I think.
*


For the low anchor, I suggest using something that sounds bad for sure, like l3enc.
*



Atrac?
Go to the top of the page
+Quote Post
CiNcH
post Nov 12 2005, 22:29
Post #9





Group: Members
Posts: 9
Joined: 11-January 04
Member No.: 11135



ATRAC3 has certainly improved since SonicStage 2.0 (especially in 2.2 I think) but would still be the low anchor. Probably more interesting than including another MPEG layer 3 codec like l3enc. So I vote for ATRAC3 @ 132kbps with SonicStage 3.2 or 3.3.

New Nero AAC LC/HE Encoder is 4.2, not 3.0...

This post has been edited by CiNcH: Nov 12 2005, 22:43
Go to the top of the page
+Quote Post
de Mon
post Nov 12 2005, 22:42
Post #10





Group: Members
Posts: 474
Joined: 1-December 02
Member No.: 3940



QUOTE (CiNcH @ Nov 12 2005, 01:29 PM)
So I vote for ATRAC3 @ 132kbps with SonicStage 3.2 or 3.
*



If most people are against MusePack... we could really include ATRAC. With ATRAC included - the test will break TWO BIG MYTHES - about WMA and ATRAC superiority which is quite widespread in other communities.

This post has been edited by de Mon: Nov 12 2005, 23:00


--------------------
Ogg Vorbis for music and speech [q-2.0 - q6.0]
FLAC for recordings to be edited
Speex for speech
Go to the top of the page
+Quote Post
kennedyb4
post Nov 12 2005, 22:49
Post #11





Group: Members
Posts: 772
Joined: 3-October 01
Member No.: 180



QUOTE (de Mon @ Nov 12 2005, 04:42 PM)
QUOTE (CiNcH @ Nov 12 2005, 01:29 PM)
So I vote for ATRAC3 @ 132kbps with SonicStage 3.2 or 3.
*



If most people are against MusePack... we could really include ATRAC. With ATRAC included - the test will break TWO BIG MYTHES - about WMA and ATRACK superiority which is quite widespread in other communities.
*



Maybe. The last time the atrac community just attacked the testing methodology used.

Die Hards........ rolleyes.gif
Go to the top of the page
+Quote Post
skelly831
post Nov 12 2005, 22:49
Post #12





Group: Members (Donating)
Posts: 782
Joined: 11-April 05
From: México
Member No.: 21361



QUOTE (de Mon @ Nov 12 2005, 01:42 PM)
QUOTE (CiNcH @ Nov 12 2005, 01:29 PM)
So I vote for ATRAC3 @ 132kbps with SonicStage 3.2 or 3.
*



If most people are against MusePack... we could really include ATRAC. With ATRAC included - the test will break TWO BIG MYTHES - about WMA and ATRACK superiority which is quite widespread in other communities.
*


This sounds like a good idea, and at the bitrate that the test will be done, the results should be very interesting.


--------------------
we was young an' full of beans
Go to the top of the page
+Quote Post
rjamorim
post Nov 12 2005, 23:04
Post #13


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (de Mon @ Nov 12 2005, 07:42 PM)
If most people are against MusePack... we could really include ATRAC. With ATRAC included - the test will break TWO BIG MYTHES - about WMA and ATRAC superiority which is quite widespread in other communities.
*


Didn't we do it already more than an year ago?

Testing ATRAC3 is beating a dead horse, IMO, even if it improved since SS 2.0. I doubt it got much better (Sony has other things to worry about, like DRMing their format to the limit and spreading rootkits throughout the world), it's forcedly DRMd - DRM is not an option like in WMA - and even though it was bad, it wasn't bad enough to act like an anchor. The anchor is there to avoid people ranking samples too low just because there isn't anything worse, even though said samples are prefectly acceptable otherwise.


IMO, good anchors would be old or weird encoders from RRW (like l3dec, MBaacenc, QDesign, AUPECg2), or a lowpass.

This post has been edited by rjamorim: Nov 12 2005, 23:07


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
JeanLuc
post Nov 12 2005, 23:23
Post #14





Group: Members
Posts: 1311
Joined: 4-June 02
From: Cologne, Germany
Member No.: 2213



If you go with WMA, I'd suggest to go for WMA Standard since WMA Pro does not play on any portable device and thus is completely useless in a real-world-scenario.

QUOTE (rjamorim @ Nov 12 2005, 10:04 PM)
... (Sony has other things to worry about, like DRMing their format to the limit and spreading rootkits throughout the world) ...


laugh.gif


--------------------
The name was Plex The Ripper, not Jack The Ripper
Go to the top of the page
+Quote Post
Sebastian Mares
post Nov 12 2005, 23:30
Post #15





Group: Members
Posts: 3630
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



QUOTE (JeanLuc @ Nov 13 2005, 12:23 AM)
If you go with WMA, I'd suggest to go for WMA Standard since WMA Pro does not play on any portable device and thus is completely useless in a real-world-scenario.
*


I agree with that.


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
zima
post Nov 13 2005, 00:19
Post #16





Group: Members
Posts: 136
Joined: 3-July 03
From: Pomerania
Member No.: 7541



I don't - retards from Slashdot (which rjamorin mentioned) will claim that Pro isn't included because of its superiority (and that it will Soon™ be supported by portables anyway...)


--------------------
http://last.fm/user/zima
Go to the top of the page
+Quote Post
rjamorim
post Nov 13 2005, 00:46
Post #17


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (JeanLuc @ Nov 12 2005, 08:23 PM)
since WMA Pro does not play on any portable device and thus is completely useless in a real-world-scenario.
*


Erm... you could use pretty much the same argument for MPC. And that didn't keep it from being tested on my older tests.

QUOTE (zima)
I don't - retards from Slashdot (which rjamorin mentioned) will claim that Pro isn't included because of its superiority (and that it will Soon™ be supported by portables anyway...)


Indeed, I agree. F*** SlashDot. We're not testing for the clueless hordes populating that place.


I just thought of another reason not to use ATRAC3: Sebastian is a nice guy. He doesn't deserve to feel the infinite pain involved with installing and using SonicStage laugh.gif

This post has been edited by rjamorim: Nov 13 2005, 00:48


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
MuncherOfSpleens
post Nov 13 2005, 00:55
Post #18





Group: Members
Posts: 109
Joined: 25-October 05
From: Florida
Member No.: 25360



By the way, has WMA Pro ever really been compared in a listening test like this? If so, could I be provided with a link? If not, could it be included in this test? I'm curious to see how good it really is.
Go to the top of the page
+Quote Post
rjamorim
post Nov 13 2005, 00:59
Post #19


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (MuncherOfSpleens @ Nov 12 2005, 09:55 PM)
If so, could I be provided with a link?
*


http://www.rjamorim.com/test/128extension/results.html

Quite outdated though.


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
Gabriel
post Nov 13 2005, 01:11
Post #20


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



L3enc would be an interesting low anchor, as it would allow to check evolution of "state of the art" encoders. However, it might be a little too good for a low anchor.
Go to the top of the page
+Quote Post
rjamorim
post Nov 13 2005, 01:21
Post #21


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (Gabriel @ Nov 12 2005, 10:11 PM)
However, it might be a little too good for a low anchor.
*


Even 1.0?

http://www.rjamorim.com/rrw/l3enc.html


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
Dibrom
post Nov 13 2005, 01:33
Post #22


Founder


Group: Admin
Posts: 2958
Joined: 26-August 02
From: Nottingham, UK
Member No.: 1



A low anchor would be nice, but personally I would probably prefer WMA Standard and WMA Pro.

I think ATRAC would be a waste of time because:
1. It's already pretty well known by most people who actually do any real research on the topic that it's inferior to just about all modern alternatives.
2. It's fading in popularity, and will continue to do so quite rapidly in the near future.
3. The people that feel that ATRAC are superior seem quite resistant to persuasion by evidence.

So, ATRAC isn't really a worthwhile choice.

I don't think MPC is really a good choice either, because:
1. It isn't really used at this bitrate (not explicitly that is, although VBR may sometimes cause it to average out at this bitrate), even if it can perform well.
2. I don't know if it's fading in popularity, but it's not really growing either. In terms of relevancy for the results, I think MPC would be less useful for a large audience.
3. It hasn't seen any serious development in a long time, whereas other alternatives might have, and would be more interesting from a results perspective -- MPC's performance is pretty well established either way, so it could serve possibly as some sort of anchor, but beyond that it's not going to provide much new data.

I understand that a low anchor is important, especially in order to provide a reference point illustrating just how much some of the codecs in this test have improved either upon recent iterations, or older/inferior codec designs in general.

Maybe the best choice would be to add WMA Standard and WMA Pro, AND a low anchor. That has the danger of making the test too tedious and complex, but maybe it is still doable...
Go to the top of the page
+Quote Post
ErikS
post Nov 13 2005, 03:19
Post #23





Group: Members
Posts: 757
Joined: 8-October 01
Member No.: 247



I don't understand why everyone insist to inlude WMA Standard instead of the Pro version... You don't cripple other formats the same way, but wma everybody wants WMA to be bad and just to make sure it is, only test the ancient WMA Std. Ask Woodinville which version to use, and I'm pretty sure he suggest using Pro instead of Std.
Go to the top of the page
+Quote Post
Gambit
post Nov 13 2005, 03:42
Post #24


Burrrn developer


Group: Developer
Posts: 917
Joined: 25-November 01
From: Bratislava, Slovakia
Member No.: 534



I definitely would like to see both WMA Standard and Pro included. I think testing against a lower anchor can be done in an another test. You can say whatever you want about WMA, but it at least had gapless support way before some of the other codecs had it (and some still don't have it, yes, I'm looking at you Apple). And I'm very much interested in the differences between WMA Standard and Pro.


--------------------
Burrrn - http://www.burrrn.net/
MPEG Audio Collection - http://mac.sourceforge.net/
Go to the top of the page
+Quote Post
kotrtim
post Nov 13 2005, 04:30
Post #25





Group: Members
Posts: 657
Joined: 4-December 02
Member No.: 3989



QUOTE
QUOTE(JeanLuc @ Nov 13 2005, 12:23 AM)
If you go with WMA, I'd suggest to go for WMA Standard since WMA Pro does not play on any portable device and thus is completely useless in a real-world-scenario.
*



I agree with that.


I think we can drop WMA or even FAAC safely now, there's no improvements (version is still the same) since the last test, it is fine to just refer to the previous results

Unlike iTunes, Nero, LAME...they are all improved

WMA Pro vbr 128 kbps 2-pass is not in the previous test, it is interesting and more meaningful to add WMA Pro this time
Go to the top of the page
+Quote Post

35 Pages V   1 2 3 > » 
Closed TopicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 23rd September 2014 - 11:38