IPB

Welcome Guest ( Log In | Register )

3 Pages V   1 2 3 >  
Reply to this topicStart new topic
Multiformat Listening Test @ 48 kbps - FINISHED
Sebastian Mares
post Dec 9 2006, 23:17
Post #1





Group: Members
Posts: 3630
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



The much awaited results of the Public, Multiformat Listening Test @ 48 kbps are ready.

Here is the results page: http://www.maresweb.de/listening-tests/mf-48-1/results.htm



Nero is first, followed by Vorbis and WMA Pro. which are tied on second place, WMA Standard is third and loses.

I think this test shows that with modern encoders, the quality at 48 kbps is acceptable and should be good enough for Internet streaming or portable use with cell phones for example. It's also interesting to see that WMA Professional perfomed quite well although it was the only contender that used CBR.

This post has been edited by Sebastian Mares: Dec 9 2006, 23:45


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
rjamorim
post Dec 9 2006, 23:43
Post #2


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



Fantastic! Thank-you very much for your efforts, Camil tongue.gif


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
Sebastian Mares
post Dec 9 2006, 23:54
Post #3





Group: Members
Posts: 3630
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Don't mention it.

Anyways, congrats to Nero and its devs for a nice low-bitrate codec. Another interesting thing is that iTunes at 96 kbps seems to be transparent to most users.

This post has been edited by Sebastian Mares: Dec 10 2006, 00:02


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
Gambit
post Dec 10 2006, 00:05
Post #4


Burrrn developer


Group: Developer
Posts: 917
Joined: 25-November 01
From: Bratislava, Slovakia
Member No.: 534



Thanks to Sebastian and everybody who contributed.


--------------------
Burrrn - http://www.burrrn.net/
MPEG Audio Collection - http://mac.sourceforge.net/
Go to the top of the page
+Quote Post
Junon
post Dec 10 2006, 00:58
Post #5





Group: Members
Posts: 520
Joined: 27-August 06
From: Germany
Member No.: 34518



QUOTE (Sebastian Mares @ Dec 9 2006, 23:54) *
Anyways, congrats to Nero and its devs for a nice low-bitrate codec.


Though I'm even more impressed about how the new Windows Media Audio 10 Professional codec performed. As we all know, VBR can drastically increase the overall sound quality compared to plain CBR. But although the codec was in a clear disadvantage compared to its antagonists, it was still able to rank as 2nd, with its result not being too far from Nero. I wonder how that's possible - does it feature some special techniques for low-bitrate encoding, comparable to Nero's SBR?
Go to the top of the page
+Quote Post
rjamorim
post Dec 10 2006, 01:02
Post #6


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (Junon @ Dec 9 2006, 20:58) *
As we all know, VBR can drastically increase the overall sound quality compared to plain CBR.


That's arguable. Indeed, a good VBR model can improve things a lot. But a bad model can ruin everything. Check the iTunes MP3 encoder for an example.

QUOTE
does it feature some special techniques for low-bitrate encoding, comparable to Nero's SBR?


Yes


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
guruboolez
post Dec 10 2006, 01:02
Post #7





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



Great test, thanks!
Two interesting points:

• iTunes AAC CBR at 96 kbps as high anchor get the same score than iTunes AAC at VBR 128 kbps when tested as competitor.

• quality varies a lot with samples. It means that at such low bitrate VBR doesn't imply constant quality.
eig, aquatisme, spmg54 and bibilolo were all ranked under 3.0/5 with all competitors (with one exception for bibilolo = 3.16 with vorbis). On the other side other samples like locomotive breath, symphony metal, bebussy, white america, are close to transparency.
In other words, current encoders could sound pretty well but also poorly. Quality is simply unstable - and VBR doesn't really help.

This post has been edited by guruboolez: Dec 10 2006, 01:04
Go to the top of the page
+Quote Post
Redmond
post Dec 10 2006, 01:03
Post #8





Group: Members
Posts: 11
Joined: 9-December 04
Member No.: 18586



WMA Standard came out significantly better than iTunes AAC-LC at this bit rate...
Go to the top of the page
+Quote Post
rjamorim
post Dec 10 2006, 01:54
Post #9


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (Redmond @ Dec 9 2006, 21:03) *
WMA Standard came out significantly better than iTunes AAC-LC at this bit rate...


That's to be expected. iTunes AAC doesn't implement intensity stereo.


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
Junon
post Dec 10 2006, 02:06
Post #10





Group: Members
Posts: 520
Joined: 27-August 06
From: Germany
Member No.: 34518



QUOTE (Redmond @ Dec 10 2006, 01:03) *
WMA Standard came out significantly better than iTunes AAC-LC at this bit rate...


That's also a bit surprising to me, starting from the fact that people keep complaining about WMA Standard's bad quality here on HA, although Apple's AAC implementation performs even worse. Might be possible that it's the other way round on higher quality settings. But nonetheless, that doesn't change the fact that they're both quite impractical in the 48 kbps bitrate region. During the past few years I've been using WMA 9 STD Q50 on two portable devices which supported nothing but MP3 and WMA, and that was the first bearable compromise between filesize and quality.

Thanks to Roberto for the quick reply about WMA 10 Pro. At the first moment I wondered a bit about my lossy codec's of choice result, Vorbis, which is just on par with that CBR competitor made by Microsoft. But since both Nero and Microsoft added low-bitrate techniques to their codecs (SBR in Nero's case, don't know what Microsoft exactly uses in WMA Pro) in contrast to Vorbis, I'd say aoTuV beta 5 did a good job in this test.

Edit: Ah, the answer about WMA STD vs. iTunes AAC can be found above. Should've reloaded this topic before replying.

This post has been edited by Junon: Dec 10 2006, 02:08
Go to the top of the page
+Quote Post
Firon
post Dec 10 2006, 02:35
Post #11





Group: Members
Posts: 830
Joined: 3-November 05
Member No.: 25526



Boy, I don't remember giving such high scores to a lot of those samples... lalala.gif

I'm impressed with iTunes at 96kbps though. I'm definitely impressed with Nero, it does an excellent job at 48kbps (well, considering the bitrate anyway...).

I also expected Vorbis to do worse than it did.

TomsDiner was awful on all encoders though (my highest rating was a 2), save for iTunes at 96; I don't know how it got such a high average. tongue.gif


Sebastian Mares: are you planning to do any other listening tests? Maybe one at 96kbps, or god forbid, one at 24? smile.gif 96kbps might be a little difficult since it's already apparently transparent for a lot of people, but it would be nice to test, since I think with Vorbis and AAC, that's the new minimum bar (instead of 128kbps) for transparency.

This post has been edited by Firon: Dec 10 2006, 02:47
Go to the top of the page
+Quote Post
Mangix
post Dec 10 2006, 03:05
Post #12





Group: Members
Posts: 587
Joined: 26-February 06
Member No.: 28077



QUOTE (Firon @ Dec 9 2006, 17:35) *
I also expected Vorbis to do worse than it did.

opposite for me. at ~96kbps, Vorbis does a better job than HE-AAC IMO. i've only tested it with 2 songs but Vorbis still outperformed HE-AAC
Go to the top of the page
+Quote Post
IgorC
post Dec 10 2006, 03:15
Post #13





Group: Members
Posts: 1576
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Thank you for result. I found my results in anonumous37xxx.txt smile.gif

But some *.txt files haven't any ABX results. Only marks. It's seems to be normal. Why?

I waıted somethıng more from WMA 10 pro at 48. More close to Nero than to Vorbıs.

there is a gram. error in name of second sample. It's sample was uploaded by me and I taped it bad. Sample's name should be symphony_metal (my paranoia doesn't admit spaces either). More correctly saying the song is from S&M Metallıca live album called " Wherever I May Roam".

This post has been edited by IgorC: Dec 10 2006, 03:49
Go to the top of the page
+Quote Post
Jillian
post Dec 10 2006, 03:47
Post #14





Group: Members
Posts: 66
Joined: 29-April 06
From: Thailand
Member No.: 30166



Could you give me any hints of the next listening test?
Please crying.gif (This test doesn't surprise me at all)
Go to the top of the page
+Quote Post
Redmond
post Dec 10 2006, 04:33
Post #15





Group: Members
Posts: 11
Joined: 9-December 04
Member No.: 18586



QUOTE (IgorC @ Dec 9 2006, 18:15) *
I waıted somethıng more from WMA 10 pro at 48. More close to Nero than to Vorbıs.


Does Nero use PS (HE-AAC v2) at this quality?
Go to the top of the page
+Quote Post
Mangix
post Dec 10 2006, 04:47
Post #16





Group: Members
Posts: 587
Joined: 26-February 06
Member No.: 28077



QUOTE (Redmond @ Dec 9 2006, 19:33) *
QUOTE (IgorC @ Dec 9 2006, 18:15) *

I waıted somethıng more from WMA 10 pro at 48. More close to Nero than to Vorbıs.


Does Nero use PS (HE-AAC v2) at this quality?

no
Go to the top of the page
+Quote Post
singaiya
post Dec 10 2006, 04:55
Post #17





Group: Members
Posts: 365
Joined: 21-November 02
Member No.: 3830



edit: Nevermind, I figured it out. Thank you for running the test, Sebastian.

This post has been edited by singaiya: Dec 10 2006, 05:05
Go to the top of the page
+Quote Post
singaiya
post Dec 10 2006, 05:57
Post #18





Group: Members
Posts: 365
Joined: 21-November 02
Member No.: 3830



In a quick comparison, I found that my results more or less match those of the cumulative (imagine that). The only variance was choosing 2nd place. On some samples I ranked Vorbis higher where the group total ranked WMA Pro higher, and vice versa on others. Of course, it's so close that the <95% explains that.

One interesting find was on the debussy sample I ranked the low anchor 3.8, and WMA Std 1.0. Also on Kraftwerk I ranked vorbis 1.0 and low anchor 2.7.

No encoder handled biblio very well.

I'll share this discovery I had on sample 18 (the wizard). From my results log: "At this sample I notice that some encoders are doing well at certain parts of the sample and worse at others. If I focus on a part in the beginning, quickly put the sliders to a rough area, then focus on some other part (in order to fine tune the ratings), the ratings from the first pass often don't apply. Then we must decide which failure is more important.

For example, here encoder 3 was better than the others in the first few seconds. But much worse once the vocals came in at the end."

Did anyone else notice this at any point?
Go to the top of the page
+Quote Post
ff123
post Dec 10 2006, 06:01
Post #19


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



QUOTE (IgorC @ Dec 9 2006, 18:15) *
Thank you for result. I found my results in anonumous37xxx.txt smile.gif

But some *.txt files haven't any ABX results. Only marks. It's seems to be normal. Why?


ABX testing is optional in these tests. Perhaps that should be made more clear in the instructions, if it isn't already.

Thanks for the interesting test, Sebastian. Looks like lots of people were interested.
Go to the top of the page
+Quote Post
IgorC
post Dec 10 2006, 06:53
Post #20





Group: Members
Posts: 1576
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



QUOTE (ff123 @ Dec 9 2006, 21:01) *
ABX testing is optional in these tests. Perhaps that should be made more clear in the instructions, if it isn't already.

Can you gıve, please, dırect lınk where it says that.

This post has been edited by IgorC: Dec 10 2006, 06:57
Go to the top of the page
+Quote Post
ff123
post Dec 10 2006, 07:30
Post #21


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



QUOTE (IgorC @ Dec 9 2006, 21:53) *
QUOTE (ff123 @ Dec 9 2006, 21:01) *

ABX testing is optional in these tests. Perhaps that should be made more clear in the instructions, if it isn't already.

Can you gıve, please, dırect lınk where it says that.


It's not directly stated what is required and what is optional.

Sebastian's readme file says to consult this page:

http://ff123.net/64test/practice.html

That tutorial should probably be updated to at least show screenshots of abchr-java. ABX is emphasized so much on HA that it may come as a surprise to some people that not all double-blind tests require this type of repeated result.

ff123
Go to the top of the page
+Quote Post
Sebastian Mares
post Dec 10 2006, 08:41
Post #22





Group: Members
Posts: 3630
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



This was stated in the listening test thread: as long as there are no ranked references, a simple score is enough and ABX test can be omitted.

By the way, for people who want to modify plots: http://www.maresweb.de/listening-tests/mf-48-1/pandts.xls

This post has been edited by Sebastian Mares: Dec 10 2006, 08:39


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
jido
post Dec 10 2006, 11:29
Post #23





Group: Members
Posts: 246
Joined: 10-February 04
From: London
Member No.: 11923



QUOTE (Sebastian Mares @ Dec 9 2006, 14:17) *
The much awaited results of the Public, Multiformat Listening Test @ 48 kbps are ready.

Here is the results page: http://www.maresweb.de/listening-tests/mf-48-1/results.htm


Thanks for the test and all participants for the results!

The link printed in the graph is broken.
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Dec 10 2006, 12:06
Post #24


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



QUOTE (Junon @ Dec 9 2006, 23:58) *
QUOTE (Sebastian Mares @ Dec 9 2006, 23:54) *
Anyways, congrats to Nero and its devs for a nice low-bitrate codec.


Though I'm even more impressed about how the new Windows Media Audio 10 Professional codec performed. As we all know, VBR can drastically increase the overall sound quality compared to plain CBR. But although the codec was in a clear disadvantage compared to its antagonists, it was still able to rank as 2nd, with its result not being too far from Nero.


I wouldn't be so sure about almost mythical powers of VBR at such low bit rate.

For example, VBR does not do any wonder for Nero HE-AAC @48 kbps:

http://www.hydrogenaudio.org/forums/index....showtopic=41191

As you can see, for Nero HE-AAC 48 kbps, CBR and VBR were pretty much the same, VBR being marginally better. And, Nero's VBR implementation is pretty good as it could be seen from this test (German Speech sample - average bit rate 31 kbps - and the codec was still performing best on average - even compared to much higher bit rate contenders)

So, I strongly doubt that usage of VBR in WMA codec, even if it could be forced to give average 48 kbps (which was not possible) would change the score so much, that it would make statistical difference.
Go to the top of the page
+Quote Post
Raptus
post Dec 10 2006, 12:19
Post #25





Group: Members
Posts: 73
Joined: 22-February 04
From: Germany
Member No.: 12191



Very interesting results. Not that I have use for such low bitrates, but very impressive performance by some.

Ah, found myself as anonymous12 cool.gif
Go to the top of the page
+Quote Post

3 Pages V   1 2 3 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 22nd September 2014 - 02:21