IPB

Welcome Guest ( Log In | Register )

2 Pages V   1 2 >  
Reply to this topicStart new topic
NSTL WMA Pro listening test, MS claims test proves WMA Pro > HE-AAC
Woodinville
post Jan 12 2006, 01:01
Post #1





Group: Members
Posts: 1402
Joined: 9-January 05
From: JJ's office.
Member No.: 18957



QUOTE ([JAZ] @ Jan 11 2006, 12:40 PM)
QUOTE (gameplaya15143 @ Jan 11 2006, 03:32 AM)

i think they are 'supposed' to represent a % quality.. but they are just high-to-low (or should it say low-to-lower... uh oh.. i better watch out for tos8)
*



Have you heard problems with a ~240Kbps VBR (Q98) wma 9.1 file to say it is low quality?
*



How about this new report (DBT and all) on WMA Pro quality?

http://www.microsoft.com/windows/windowsme...Comparison.aspx


--------------------
-----
J. D. (jj) Johnston
Go to the top of the page
+Quote Post
Cyaneyes
post Jan 12 2006, 02:05
Post #2





Group: Members
Posts: 297
Joined: 21-September 03
Member No.: 8934



QUOTE (Woodinville @ Jan 11 2006, 07:01 PM)
How about this new report (DBT and all) on WMA Pro quality?

http://www.microsoft.com/windows/windowsme...Comparison.aspx
*



It would be nice if they stated the bitrate of the HE-AAC used in the test.
Go to the top of the page
+Quote Post
gameplaya15143
post Jan 12 2006, 02:33
Post #3





Group: Members
Posts: 484
Joined: 8-January 06
From: Earth
Member No.: 26978



QUOTE (Cyaneyes @ Jan 11 2006, 08:05 PM)
It would be nice if they stated the bitrate of the HE-AAC used in the test.
*
one would asume that the bitrate was close to that of the wma it was being compared to... unsure.gif but that IS just an asumption

i liked this line: "71% of all listeners indicated that WMA Pro was equal to or better than HE AAC."

there is a 48kbps multiformat test coming up isnt there? I'll wait for those results smile.gif

@[JAZ] no problems at that kind of a bitrate.. but then again i dont know, i don't really use wma, i just keep the dbpoweramp encoder around for occational tests wink.gif


--------------------
Vorbis-q0-lowpass99
lame3.93.1-q5-V9-k-nspsytune
Go to the top of the page
+Quote Post
stephanV
post Jan 12 2006, 12:34
Post #4





Group: Members
Posts: 394
Joined: 6-May 04
Member No.: 13932



QUOTE (Cyaneyes @ Jan 12 2006, 02:05 AM)
It would be nice if they stated the bitrate of the HE-AAC used in the test.
*

Bitrate is in the pdf and is indeed 64 kbps. Even precise steps to how the clips were encoded are given in there.

Of course using the same rethoric you could say that 63% of the participants rated HE-AACv2 equal or better than WMA.


--------------------
"We cannot win against obsession. They care, we don't. They win."
Go to the top of the page
+Quote Post
kritip
post Jan 12 2006, 13:10
Post #5





Group: Members
Posts: 528
Joined: 15-January 02
From: Warwickshire -- England
Member No.: 1036



Anyone notive the MS codec used?

QUOTE
15. Set the “Codec” to Windows Media Audio 10 Professional+


I don't keep up on the WMA codecs, is this a new version?? I thought 9.1 Pro wsa the latest avaliable??

Kristian
Go to the top of the page
+Quote Post
[JAZ]
post Jan 12 2006, 19:03
Post #6





Group: Members
Posts: 1783
Joined: 24-June 02
From: Catalunya(Spain)
Member No.: 2383



** Test codecs:

WMA Pro at 64Kbps? Interesting...It might be the codec they are making for Vista.
Let's see if now they ditch WMA Standard (or start to do so, even though "PlaysForSure")

OTOH, wouldn't HE-AAC V1 achieve better results than HE-AAC v2 at 64kbps?

**Test methodology:
Faults ( IMO ):

From the .pdf, it seems that nor ABX neither ABC-HR was done when electing the samples(1), plus the randomness in which the samples where played is questionable(2).

Overall, most of the results seem random(3), except for a few samples(4) where there could really be a difference.


(1) "The first clip to be played for every song was the reference clip. After listening to the reference clip, the participant would hear two additional clips, Clip-A and Clip-B. The participant was asked to compare both Clip-A and Clip-B to the reference file, and determine which sounded more like the reference file. NSTL would replay any of the three clips for the participant upon request, but once a decision was made on a song, the test would advance to the next song"

(2) BBABBBAAABAA ( B->HE-AAC first, A-> WMA first. ). They did this for 2/3rd of the participants, and inverted the order for the other 1/3rd.


(3) For most of the samples, the listeners elected "Identical". The percentage in which they elected wma or he-aac is similar. No big differences in prefference (only ~5% although generally is in favour of WMA...)

(4) The results, from my POV (point of view):
BobMarley,ColdPlay,Greenday,si02 -> Favour WMA

SimpleMinds,BlankBaby,Floyd -> Seem to Favour WMA, but with a small margin.

LimpBizkid,DireStraits -> There really is a difference, but prefference between one or the other is random (or averaged)

GuanoApes,Maroon5, -> There could be a difference, but the distribution is much averaged.

TheVerve -> Seem to favour HE-AAC, but with a small margin.


**Conclusion:

It could be true that WMA Pro 10 at 64Kbps is better than Nero 7, HE-AAC v2 at 64Kbps from the results shown. It definitely shows some problematic samples (especifically BobMarley,Greenday,si02) for HE-AAC, and that WMA Pro 10 is now competitive at this bitrate.
Let's see if it can be included in the multiformat low-bitrate test that we will do soon.



[Edit -> Forgot to say a few things]

This post has been edited by [JAZ]: Jan 12 2006, 19:23
Go to the top of the page
+Quote Post
spoon
post Jan 12 2006, 20:59
Post #7


dBpowerAMP developer


Group: Developer (Donating)
Posts: 2748
Joined: 24-March 02
Member No.: 1615



> don't keep up on the WMA codecs, is this a new version?? I thought 9.1 Pro wsa the latest avaliable??

9.1 is the latest, the confusion comes from WMP 10.

This report, to tell you the truth I wouldn't believe 1 word of it, even if it appears to be from a separate testing agency (guess who pay their bills...). The same goes for any company, Nero, FhG, any self-tests or sponsored tests should not be given a second thought, it is all advertising spiel. That is why listening tests by non-affiliated groups / people are worth their weight in gold.


--------------------
Spoon http://www.dbpoweramp.com
Go to the top of the page
+Quote Post
singaiya
post Jan 12 2006, 21:49
Post #8





Group: Members
Posts: 365
Joined: 21-November 02
Member No.: 3830



I found it interesting that at the very bottom of the pdf report, they write "NSTL does not guarantee the accuracy, adequacy or completeness of the services provided to Microsoft or the data included herein." I know that disclaimers are common, but in this case it seems ironic especially for a company that on the same page claims "NSTL is the leading independent information technology testing organization for the computer industry, dedicated to providing high quality services and test tools to hardware developers, software publishers, government agencies and corporations."

Also for such an large testing group I think they should have included more types of music such as classical and jazz. It's nearly all rock/pop oriented.
Go to the top of the page
+Quote Post
gameplaya15143
post Jan 13 2006, 04:57
Post #9





Group: Members
Posts: 484
Joined: 8-January 06
From: Earth
Member No.: 26978



QUOTE (singaiya @ Jan 12 2006, 03:49 PM)
"NSTL is the leading independent information technology testing organization for the computer industry, dedicated to providing high quality services and test tools...."
*
.... they forgot to add "maybe" laugh.gif
I didn't know they were using some new wma 10+ codec... just have to wait till it's available to see how it really fares rolleyes.gif


--------------------
Vorbis-q0-lowpass99
lame3.93.1-q5-V9-k-nspsytune
Go to the top of the page
+Quote Post
Woodinville
post Jan 14 2006, 08:02
Post #10





Group: Members
Posts: 1402
Joined: 9-January 05
From: JJ's office.
Member No.: 18957



I must admit that I am both offended and annoyed by the various insinuations in this thread. Hence, I shall say nothing here, and await evidence of the various insinuations and suggestions.


--------------------
-----
J. D. (jj) Johnston
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Jan 14 2006, 09:32
Post #11


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



@Woodinville, is there any way for the public to obtain the codec used in the test - so some kind of public verification could be done in the future, or this codec is to be released with Vista?

There is a new 48 kbps multiformat listening test scheduled for, most likely, March (after 48 kbps AAC listening test) - and testing the latest WMA codec would be a very very nice thing, especially after latest claims from the NSTL.

This post has been edited by Ivan Dimkovic: Jan 14 2006, 09:42
Go to the top of the page
+Quote Post
Garf
post Jan 14 2006, 10:25
Post #12


Server Admin


Group: Admin
Posts: 4885
Joined: 24-September 01
Member No.: 13



QUOTE (Woodinville @ Jan 14 2006, 09:02 AM)
I must admit that I am both offended and annoyed by the various insinuations in this thread. Hence, I shall say nothing here, and await evidence of the various insinuations and suggestions.
*


I'll make 2 factual observations:

1) If I follow the test procedure, the Nero files will be 64kbps LC-AAC, not HE-AAC. In their very detailed description, they simply didn't enable HE-AAC in the settings dialog. You can easily verify this with Nero 7 (even the free demo version should work).

Edit: Since this behaviour seems to have been fixed/may not have existed in the version used, you can ignore this for the sake of argument. I'll replace it by: 1) It seems impossible to verify the test results and setup.

2) They consulted with MS how to set up WMA and got new tools that are apparently not publicly available. They didn't consult with us, and used an old version of the Nero AAC encoder.

This post has been edited by Garf: Jan 14 2006, 16:02
Go to the top of the page
+Quote Post
Gabriel
post Jan 14 2006, 13:29
Post #13


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



QUOTE
1) If I follow the test procedure, the Nero files will be 64kbps LC-AAC, not HE-AAC. In their very detailed description, they simply didn't enable HE-AAC in the settings dialog. You can easily verify this with Nero 7 (even the free demo version should work).

You mean that when selecting “Nero Digital Audio (HE-AAC v2) (*.mp4)” only AAC-LC is used by default? If this is the case, you probably have a user interface design problem.
Go to the top of the page
+Quote Post
Latexxx
post Jan 14 2006, 14:01
Post #14


A/V Moderator


Group: Members
Posts: 858
Joined: 12-May 03
From: Finland
Member No.: 6557



QUOTE (Garf @ Jan 14 2006, 11:25 AM)
1) If I follow the test procedure, the Nero files will be 64kbps LC-AAC, not HE-AAC. In their very detailed description, they simply didn't enable HE-AAC in the settings dialog. You can easily verify this with Nero 7 (even the free demo version should work).
*

If this is the case, somebody should contact Microsoft because they can't keep that comparison online if they know that it is completely false. Of course they could still keep it on their page but that would likely be illegel if they keep claiming that the paper proves wma's superiority to he aac.

And doesn't that mean that Microsoft's hyper super ultra new and amazing WMA 10 Pro + is only tied with lc aac instead of he aac.

This post has been edited by Latexxx: Jan 14 2006, 14:03
Go to the top of the page
+Quote Post
guruboolez
post Jan 14 2006, 14:09
Post #15





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (Latexxx @ Jan 14 2006, 02:01 PM)
Of course they could still keep it on their page but that would likely be illegel if they keep claiming that the paper proves wma's superiority to he aac.
*

Illegal? Do you mean that results of a test shouldn't be revealed if one term of the comparison is buggy?
Go to the top of the page
+Quote Post
Latexxx
post Jan 14 2006, 14:28
Post #16


A/V Moderator


Group: Members
Posts: 858
Joined: 12-May 03
From: Finland
Member No.: 6557



QUOTE (guruboolez @ Jan 14 2006, 03:09 PM)
QUOTE (Latexxx @ Jan 14 2006, 02:01 PM)
Of course they could still keep it on their page but that would likely be illegel if they keep claiming that the paper proves wma's superiority to he aac.
*

Illegal? Do you mean that results of a test shouldn't be revealed if one term of the comparison is buggy?
*


No. But if they really have tested lc aac and know it, they can't keep lying by keeping that document publically available.
Go to the top of the page
+Quote Post
Garf
post Jan 14 2006, 14:31
Post #17


Server Admin


Group: Admin
Posts: 4885
Joined: 24-September 01
Member No.: 13



QUOTE (Gabriel @ Jan 14 2006, 02:29 PM)
QUOTE
1) If I follow the test procedure, the Nero files will be 64kbps LC-AAC, not HE-AAC. In their very detailed description, they simply didn't enable HE-AAC in the settings dialog. You can easily verify this with Nero 7 (even the free demo version should work).

You mean that when selecting “Nero Digital Audio (HE-AAC v2) (*.mp4)” only AAC-LC is used by default? If this is the case, you probably have a user interface design problem.
*



You select that "Nero Digital Audio (HE-AAC)" out of a list of MP3Pro, TwinVQ, WMA, AIFF, WAV etc.

By default, it will encode to 128kbps LC-AAC after selecting that.

Selecting the CBR/Stereo - 64kbps [HE-AAC] preset works correctly.

You have to go inside the encoder settings, force it to 64kbps CBR, and ignore the explicit combobox that says "LC (Low Complexity) AAC", to get it to fail this way. And that is exactly what their description says they did.

PS. I do agree this part of the GUI could be done better, but it can only fail this way by tinkering with some "advanced" settings and ignoring others. I don't think removing the LC-AAC options is good, either.

This post has been edited by Garf: Jan 14 2006, 14:35
Go to the top of the page
+Quote Post
guruboolez
post Jan 14 2006, 14:31
Post #18





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (Latexxx @ Jan 14 2006, 02:28 PM)
QUOTE (guruboolez @ Jan 14 2006, 03:09 PM)
QUOTE (Latexxx @ Jan 14 2006, 02:01 PM)
Of course they could still keep it on their page but that would likely be illegel if they keep claiming that the paper proves wma's superiority to he aac.
*

Illegal? Do you mean that results of a test shouldn't be revealed if one term of the comparison is buggy?
*


No. But if they really have tested lc aac and know it, they can't keep lying by keeping that document publically available.
*


I agree with you. It would be a lie. But in this case, the conducer or the laboratory has clearly set Nero to use HE-AAC. There's apparently a bug in the software, and the conducer can't be responsible of this (and can't necessary notice it while performing the test).

EDIT: emphasis in the quote

This post has been edited by guruboolez: Jan 14 2006, 14:33
Go to the top of the page
+Quote Post
Garf
post Jan 14 2006, 14:39
Post #19


Server Admin


Group: Admin
Posts: 4885
Joined: 24-September 01
Member No.: 13



QUOTE (guruboolez @ Jan 14 2006, 03:31 PM)
But in this case, the conducer or the laboratory has clearly set Nero to use HE-AAC. There's apparently a bug in the software, and the conducer can't be responsible of this (and can't necessary notice it while performing the test).
*


If you fiddle around in the settings dialog, change some settings, and it clearly says "LC (Low Complexity) AAC", how on earth can you say that "it is clearly set to HE-AAC"?

Where is the bug???
Go to the top of the page
+Quote Post
sehested
post Jan 14 2006, 14:53
Post #20





Group: Members (Donating)
Posts: 325
Joined: 5-April 04
From: Copenhagen, Denmark
Member No.: 13246



QUOTE (Garf @ Jan 14 2006, 05:39 AM)
QUOTE (guruboolez @ Jan 14 2006, 03:31 PM)
But in this case, the conducer or the laboratory has clearly set Nero to use HE-AAC. There's apparently a bug in the software, and the conducer can't be responsible of this (and can't necessary notice it while performing the test).
*


If you fiddle around in the settings dialog, change some settings, and it clearly says "LC (Low Complexity) AAC", how on earth can you say that "it is clearly set to HE-AAC"?

Where is the bug???
*

You could contact NTSL to obtain the .wav files used in the test - both reference, A, and B samples.
You could then encode the reference using the same software and settings as described by NTSL and make one LC-AAC and one HC-ACC.
This would allow you to find out what settings where in fact used.
Go to the top of the page
+Quote Post
sehested
post Jan 14 2006, 14:57
Post #21





Group: Members (Donating)
Posts: 325
Joined: 5-April 04
From: Copenhagen, Denmark
Member No.: 13246



What about volume leveling?

With the test procedure described by NTSL the encoder raising the volume the most would normally win such a comparison.

Not that I suspect neither WMA PRO nor AAC of boosting the volume, however software is software and IMHO the testing metholody should have taken this issue into account. At least provide an analysis of the gain of the resulting wav files.
Go to the top of the page
+Quote Post
Alex B
post Jan 14 2006, 15:16
Post #22





Group: Members
Posts: 1303
Joined: 14-September 05
From: Helsinki, Finland
Member No.: 24472



Hmm...

I followed the procedure described in the test report and got this Nero encoding setting:





Isn't this a HE setting?


--------------------
http://listening-tests.freetzi.com
Go to the top of the page
+Quote Post
guruboolez
post Jan 14 2006, 15:20
Post #23





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (Garf @ Jan 14 2006, 02:39 PM)
QUOTE (guruboolez @ Jan 14 2006, 03:31 PM)
But in this case, the conducer or the laboratory has clearly set Nero to use HE-AAC. There's apparently a bug in the software, and the conducer can't be responsible of this (and can't necessary notice it while performing the test).
*


If you fiddle around in the settings dialog, change some settings, and it clearly says "LC (Low Complexity) AAC", how on earth can you say that "it is clearly set to HE-AAC"?

Where is the bug???
*


You should read the testing procedure again. Step#6:
Select "Nero Digital Audio (HE-AAC v.2)....
The person has set Nero to use HE-AAC v.2 as described in the frontend. If the result is LC-AAC, then it would rather look to a bug than a feature.
Go to the top of the page
+Quote Post
Garf
post Jan 14 2006, 15:23
Post #24


Server Admin


Group: Admin
Posts: 4885
Joined: 24-September 01
Member No.: 13



QUOTE (Alex B @ Jan 14 2006, 04:16 PM)
Isn't this a HE setting?
*


Yes. Does it automatically switch to HE-AAC for you? Older versions of Nero 7 don't seem to, and one newer one I have here doesn't either.

This post has been edited by Garf: Jan 14 2006, 15:25
Go to the top of the page
+Quote Post
Garf
post Jan 14 2006, 15:24
Post #25


Server Admin


Group: Admin
Posts: 4885
Joined: 24-September 01
Member No.: 13



QUOTE (guruboolez @ Jan 14 2006, 04:20 PM)
You should read the testing procedure again. Step#6:
Select "Nero Digital Audio (HE-AAC v.2)....
The person has set Nero to use HE-AAC v.2 as described in the frontend. If the result is LC-AAC, then it would rather look to a bug than a feature.
*


I read the testing procedure and it says that person also went on to the detailed encoder settings and changed them. As I already said, using the defaults, or using a preset, all would have produced a good result.

This post has been edited by Garf: Jan 14 2006, 15:28
Go to the top of the page
+Quote Post

2 Pages V   1 2 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 21st September 2014 - 06:23