IPB

Welcome Guest ( Log In | Register )

8 Pages V  « < 6 7 8  
Reply to this topicStart new topic
Multiformat Listening Test @ 128 kbps - FINISHED
guruboolez
post Jan 23 2006, 14:52
Post #176





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (QuantumKnot @ Jan 18 2006, 12:50 AM)
I haven't been following developments in aoTuV lately, but has Aoyumi improved the block switching algorithm to reduce smearing on microattacks?  If not, then I might have a look at it again if I have time.  But if it's been fixed, then I won't have to worry. biggrin.gif
*

There's still headroom for progress I'd say. During my last listening evaluation, Vorbis was still unsharp and noisy on micro-attacks/short-impulses samples. Most often bitrate doesn't go really high on such samples (Vorbis tends to inflate the bitrate when needed - but not here).
Try with this sample for a good start smile.gif
Go to the top of the page
+Quote Post
kurtnoise
post Feb 16 2006, 08:38
Post #177





Group: Members
Posts: 326
Joined: 26-June 02
From: Aix-en-Provence
Member No.: 2400



Some statistical analysis by AMTuring about this listening test...


@Guru:: very nice your avatar btw... smile.gif


--------------------
http://www.unite-video.com/phpbb/viewtopic.php?t=5412 :: An overview of all lossless Audio Formats (in french language ;-)
Go to the top of the page
+Quote Post
Garf
post Feb 16 2006, 10:04
Post #178


Server Admin


Group: Admin
Posts: 4882
Joined: 24-September 01
Member No.: 13



QUOTE (kurtnoise @ Feb 16 2006, 09:38 AM)
Some statistical analysis by AMTuring about this listening test...
*


Oh please, can we keep the crackpot science out of this forum? This person wasn't banned here for no reason. Just juggling around scientific words doesn't magically make anything you say sensible, lest alone correct.

QUOTE
It is well known that some songs are more difficult to encode than others, and they result in lower quality encoded files regardless of the encoder used. So the assumption of equal means amongst experiments is violated.


Bzzzt. This was a VBR test. Meaning, although the average was 128kbps (or slightly more), the codecs could spend as much bits as necessary to keep all clips at a constant quality. This means you cannot immediately assume the means aren't equal, in fact it should be the opposite.

So, what happens if we actually look at the data? (note that he provides many graphcs to 'illustrate' his points, except the ones where, well, the data doesn't support his claims anywhere) The variance of the means of the samples is much less than the difference between the codecs themselves. (exluding Shine, which is CBR)

In other words, VBR works. I would have thought that that was "well known" by now.

QUOTE
The following table shows the Tukey HSD applied to the ranks.


Say what? You cannot apply plain Tukey HSD to rank scores, it's a parametric test. Now, I'm willing to argue that we shouldn't use parametric analysis (because the top end of the results clips at 5.0, and you can see this by observing that the lower rated the codec, the higher the variance). However, if anything parametric analysis gives stronger results. If you use rank scores, let's actually use the rank score version of Tukey HSD to analyze the results:

CODE

FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Nonparametric Tukey HSD analysis

Number of listeners: 18
Critical significance: 0.05
Nonparametric Tukey's HSD: 25.894

Ranksums:

Vorbis iTunes WMA Nero LAME
73.00 62.50 49.50 48.50 36.50

-------------------------- Difference Matrix --------------------------

iTunes WMA Nero LAME
Vorbis 10.500 23.500 24.500 36.500*
iTunes 13.000 14.000 26.000*
WMA 1.000 13.000
Nero 12.000
-----------------------------------------------------------------------

Vorbis is better than LAME
iTunes is better than LAME


Gee, where did those "extra" conclusions go?
Let's compare this to the means with parametric Tukey HSD:

CODE

FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Tukey HSD analysis

Number of listeners: 18
Critical significance: 0.05
Tukey's HSD: 0.110

Means:

Vorbis iTunes WMA Nero LAME
4.79 4.74 4.70 4.68 4.60

-------------------------- Difference Matrix --------------------------

iTunes WMA Nero LAME
Vorbis 0.049 0.090 0.106 0.193*
iTunes 0.041 0.056 0.143*
WMA 0.016 0.103
Nero 0.087
-----------------------------------------------------------------------

Vorbis is better than LAME
iTunes is better than LAME


Coincidence? Hardly. If you derive the rank scores from the means, how can you expect a different conclusion? How would you expect throwing away information to increase the significance? It won't, unless you use a completely wrong analysis method.

This post has been edited by Garf: Feb 16 2006, 10:23
Go to the top of the page
+Quote Post
bug80
post Feb 16 2006, 11:00
Post #179





Group: Members
Posts: 398
Joined: 23-January 05
From: The Netherlands
Member No.: 19254



Maybe it is a good idea to post your reply on Doom9 also, Garf.
Go to the top of the page
+Quote Post
ff123
post Feb 16 2006, 14:56
Post #180


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



I posted a reply on doom9, suggesting he use bootstrap resampling if he's hard up to analyze the data some more.

ff123
Go to the top of the page
+Quote Post
detokaal
post Feb 16 2006, 17:45
Post #181





Group: Members
Posts: 191
Joined: 9-November 03
Member No.: 9748



QUOTE (Garf @ Feb 16 2006, 04:04 AM)
QUOTE (kurtnoise @ Feb 16 2006, 09:38 AM)
Some statistical analysis by AMTuring about this listening test...
*

Bzzzt.


I like this biggrin.gif
Go to the top of the page
+Quote Post
torok
post Mar 22 2006, 19:35
Post #182





Group: Members
Posts: 391
Joined: 24-December 02
From: Eugene, OR
Member No.: 4224



Did anyone else notice that when you normalize for bitrate, you get this:

iTunes = 4.83
AoTuV = 4.55
lame = 4.5
WMA = 4.84

Which looks like it makes iTunes and WMA tied for first and the rest in second. Now, I'm not sure if it's reasonable to normalize like that (it's asssuming that quality of a codec is directly 1:1 with bitrate), but it's an interesting thought.


--------------------
http://www.pkulak.com
Go to the top of the page
+Quote Post
Busemann
post Mar 22 2006, 19:37
Post #183





Group: Members
Posts: 730
Joined: 5-January 04
Member No.: 10970



QUOTE (torok @ Mar 22 2006, 10:35 AM)
Did anyone else notice that when you normalize for bitrate, you get this:

iTunes = 4.83
AoTuV = 4.55
lame = 4.5
WMA = 4.84

Which looks like it makes iTunes and WMA tied for first and the rest in second. Now, I'm not sure if it's reasonable to normalize like that (it's asssuming that quality of a codec is directly 1:1 with bitrate), but it's an interesting thought.
*


The only fair way to test this would be to use CBR.
Go to the top of the page
+Quote Post
Mardel
post Sep 24 2011, 22:51
Post #184





Group: Members
Posts: 27
Joined: 12-March 08
Member No.: 51986



Hi!

I look at the 128 kbps listening test on your site, and i cant accept that aoTuV vorbis worse than aac.
And then I realize that vorbis or aac q settings are bad. First thing that is the vorbis use comma and not dot for decimal.
Second thing the ogg file is smaller than aac approx. 0,5 - 0,9 MB lesser.
Then i looked the average bitrates in foobar2000 the ogg's bitrate 20 kbit lesser than the aac.

e.g.: 158 kbps vs 138 kpbs

I encoded same track and another track with vorbis -q 4,99 settings and the bitrate difference was 1-2 kbit and the file size difference was 0-0,1 MB.

Overall your tests doesn't give a direction for which codec is better if the codec settings are wrong.

This post has been edited by db1989: Sep 25 2011, 11:35
Reason for edit: merged in from unrelated thread


--------------------
Wavpack -hh or TAK -pMax
OggVorbis aoTuVb6.03 -q 4
Go to the top of the page
+Quote Post
[JAZ]
post Sep 24 2011, 23:36
Post #185





Group: Members
Posts: 1751
Joined: 24-June 02
From: Catalunya(Spain)
Member No.: 2383



The vorbis commandline uses comma or dot depending on the localization of the OS where it is running.
As you've found, if the setting being used was wrong, the bitrate would have been much different.
Go to the top of the page
+Quote Post
db1989
post Sep 25 2011, 11:30
Post #186





Group: Super Moderator
Posts: 5275
Joined: 23-June 06
Member No.: 32180



QUOTE (Mardel @ Sep 24 2011, 22:51) *
Second thing the ogg file is smaller than aac approx. 0,5 - 0,9 MB lesser.
Then i looked the average bitrates in foobar2000 the ogg's bitrate 20 kbit lesser than the aac.

e.g.: 158 kbps vs 138 kpbs

Overall your tests doesn't give a direction for which codec is better if the codec settings are wrong.
Given that many modern codecs perform better, or only, in VBR or ABR modes, listening tests reporting a bitrate such as this one are generally based on the principle of tuning the codec settings to obtain that bitrate, or as close as possible to it, as the mean bitrate over the set of audio files being tested. Thus, variation between different encodes of one file are to be expected and do not invalidate the methodology (inasmuch as there doesn’t seem to be another solution; this is as close to fair as is possible).
Go to the top of the page
+Quote Post
C.R.Helmrich
post Sep 25 2011, 13:06
Post #187





Group: Developer
Posts: 686
Joined: 6-December 08
From: Erlangen Germany
Member No.: 64012



QUOTE (Mardel @ Sep 24 2011, 23:51) *
I look at the 128 kbps listening test on your site, and i cant accept that aoTuV vorbis worse than aac.

Sorry, but which test are we talking about? The test discussed in this thread is MP3 only.

Edit: Thanks lvqcl! Must be http://soundexpert.org/encoders-128-kbps then, which is from mid 2006.

Chris

This post has been edited by C.R.Helmrich: Sep 25 2011, 13:28


--------------------
If I don't reply to your reply, it means I agree with you.
Go to the top of the page
+Quote Post
lvqcl
post Sep 25 2011, 13:13
Post #188





Group: Developer
Posts: 3327
Joined: 2-December 07
Member No.: 49183



QUOTE (C.R.Helmrich @ Sep 25 2011, 16:06) *
Sorry, but which test are we talking about? The test discussed in this thread is MP3 only.


IIRC that post was moved from http://www.hydrogenaudio.org/forums/index....showtopic=77708 to this thread.
Go to the top of the page
+Quote Post
db1989
post Sep 25 2011, 13:31
Post #189





Group: Super Moderator
Posts: 5275
Joined: 23-June 06
Member No.: 32180



Thanks for pointing out my silly mistake; the irony is not lost on me… blush.gif That’s what I get for barely reading! Re(re)located to the correct thread.
Go to the top of the page
+Quote Post

8 Pages V  « < 6 7 8
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 25th July 2014 - 05:54