IPB

Welcome Guest ( Log In | Register )

3 Pages V   1 2 3 >  
Reply to this topicStart new topic
80 kbps LC AAC listening test, Are PNS and IS useful at 80 kbps?
menno
post Dec 21 2006, 11:20
Post #1


Nero MPEG4 developer


Group: Developer (Donating)
Posts: 1218
Joined: 11-October 01
From: LA
Member No.: 267



Dear All,

I would like to announce that the 80 kbps test of upcoming Nero AAC encoder has just started. The purpose of this test is to determine if PNS (Perceptual Noise Substitution) or IS (Intensity Stereo) gives any advantage when used at 80 kbps.
With this test, I wish to include true audiophile and scientific community into making the LC-AAC solutions even better in the future. Many thanks in advance for your help!
Test runs for 18 days, until January 8th 2007.

Which codecs are tested?

Following codecs are being tested:

- LC-AAC 80 kbps CBR No PNS no IS
- LC-AAC 80 kbps CBR No PNS, using IS
- LC-AAC 80 kbps CBR No IS, using PNS
- LC-AAC 80 kbps CBR Using both PNS and IS
- Hidden codec 1 80 kbps
- Hidden codec 2 (high anchor)
- Hidden codec 3 (low anchor)


How to Test

1. Download http://s4.quicksharing.com/v/5647513/ABC_HR.ZIP.html
2. Extract ABC_HR.ZIP in folder of your choice
3. Read Readme.txt from ABC_HR.ZIP
4. Download samples for the test (links are in ABC_HR.ZIP and in next post), and extract them in the directory where ABC_HR.zip was extracted
5. When you download and extract all the samples, execute the decode_all.bat files from bin folder. This will decode all the files and prepare the test
6. Now everything is ready for the test
7. Open ABC/HR by executing abchr.jar - and, to test each sample, open up the individual .ecf files. Each test set (Sample1, Sample2,) has an .ecf file for each sample
8. From now on, typical ABC/HR methodology applies, you need to judge the quality of individual files - save the results after you're done, and send the files to me, either by PP or directly to: mbakker (at) nero.com (replace at with @ to get an e-mail address)

Links to samples coming in the next post...

Happy Testing smile.gif


Here are the samples for the test. Samples from previous Mares and Robertos test at 48 kbps and 128 kbps were chosen. Test with lowest scores on those tests were chosen. This is because even at 48 kbps for many people it was hard to distinguish encoded samples from the original. Also most genres of music are present.

http://s19.quicksharing.com/v/9204458/Sample1.zip.html
http://s4.quicksharing.com/v/801115/Sample2.zip.html
http://s4.quicksharing.com/v/7815217/Sample3.zip.html
http://s4.quicksharing.com/v/3713672/Sample4.zip.html
http://s4.quicksharing.com/v/5514459/Sample5.zip.html
http://s4.quicksharing.com/v/5814487/Sample6.zip.html
http://s4.quicksharing.com/v/7761119/Sample7.zip.html
http://s4.quicksharing.com/v/3128033/Sample8.zip.html
http://s4.quicksharing.com/v/5707785/Sample9.zip.html
http://s4.quicksharing.com/v/8088598/Sample10.zip.html
http://s4.quicksharing.com/v/6741335/Sample11.zip.html
http://s4.quicksharing.com/v/1364761/Sample12.zip.html
http://s4.quicksharing.com/v/5209480/1Sample13.zip.html
http://s4.quicksharing.com/v/1426353/Sample14.zip.html
Go to the top of the page
+Quote Post
dand
post Dec 21 2006, 12:20
Post #2





Group: Members
Posts: 216
Joined: 9-December 02
From: Croatia
Member No.: 4037



Good idea to clarify the is&pns issues.

Myself: somehow sceptical on IS, some more expectation from PNS (certain type of samples only), even, as everything else, it all depends on implementation.

But lets wait fot the test results!
Go to the top of the page
+Quote Post
Sebastian Mares
post Dec 21 2006, 12:42
Post #3





Group: Members
Posts: 3629
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Can anyone confirm that downloads work only with IE?


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
menno
post Dec 21 2006, 12:45
Post #4


Nero MPEG4 developer


Group: Developer (Donating)
Posts: 1218
Joined: 11-October 01
From: LA
Member No.: 267



QUOTE (Sebastian Mares @ Dec 21 2006, 12:42) *
Can anyone confirm that downloads work only with IE?


If you block the google ads the download link doesn't show up.
Go to the top of the page
+Quote Post
Maurits
post Dec 21 2006, 12:50
Post #5





Group: Members
Posts: 386
Joined: 30-September 05
From: London, Europe
Member No.: 24805



QUOTE (Sebastian Mares @ Dec 21 2006, 11:42) *
Can anyone confirm that downloads work only with IE?

Works with Firefox 2.0.0.1 on OS X 10.4.8...
Go to the top of the page
+Quote Post
guruboolez
post Dec 21 2006, 12:59
Post #6





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



Opera 9.1 is OK too.
Go to the top of the page
+Quote Post
Sebastian Mares
post Dec 21 2006, 13:14
Post #7





Group: Members
Posts: 3629
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



QUOTE (menno @ Dec 21 2006, 12:45) *
QUOTE (Sebastian Mares @ Dec 21 2006, 12:42) *

Can anyone confirm that downloads work only with IE?


If you block the google ads the download link doesn't show up.


OK, Adblock's fault then. smile.gif


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
IgorC
post Dec 21 2006, 15:57
Post #8





Group: Members
Posts: 1573
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



menno
Only LC is included to test?
Is the reason that you want to see how PNS and IS perform on LC part only and not on SBR or new encoder will bring better quality with LC profile?
Go to the top of the page
+Quote Post
spockep
post Dec 21 2006, 19:37
Post #9





Group: Members
Posts: 120
Joined: 8-November 06
Member No.: 37320



So in short you just need the .erf files zipped then?
Go to the top of the page
+Quote Post
Firon
post Dec 22 2006, 05:47
Post #10





Group: Members
Posts: 830
Joined: 3-November 05
Member No.: 25526



Quicksharing isn't loading for me at all...
Can someone upload these to another site? Like zshare.

This post has been edited by Firon: Dec 22 2006, 05:48
Go to the top of the page
+Quote Post
menno
post Dec 22 2006, 08:45
Post #11


Nero MPEG4 developer


Group: Developer (Donating)
Posts: 1218
Joined: 11-October 01
From: LA
Member No.: 267



QUOTE (IgorC @ Dec 21 2006, 15:57) *
menno
Only LC is included to test?
Is the reason that you want to see how PNS and IS perform on LC part only and not on SBR or new encoder will bring better quality with LC profile?


The use of PNS and IS with SBR is questionable, because it will usually be used in the same range. And anyway, if PNS or IS doesn't help for LC only it will also not help when using SBR and vice versa.
Go to the top of the page
+Quote Post
halb27
post Dec 22 2006, 10:14
Post #12





Group: Members
Posts: 2435
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



Can a user expect that his LC decoder (for instance the one in my Nokia 6230) makes use of PNS and/or IS?

This post has been edited by halb27: Dec 22 2006, 10:15


--------------------
lame3100m -V1 --insane-factor 0.75
Go to the top of the page
+Quote Post
menno
post Dec 22 2006, 10:25
Post #13


Nero MPEG4 developer


Group: Developer (Donating)
Posts: 1218
Joined: 11-October 01
From: LA
Member No.: 267



QUOTE (halb27 @ Dec 22 2006, 10:14) *
Can a user expect that his LC decoder (for instance the one in my Nokia 6230) makes use of PNS and/or IS?


All AAC decoders must support IS, all MPEG-4 AAC decoders must support PNS.
To answer your question, try these samples on iPod and weep sad.gif (it's sad that Apple did not fix this yet), for other decoders the only way to find out is to test it, but besides iPod I am not aware of any uncompliant decoders.
Go to the top of the page
+Quote Post
muaddib
post Dec 22 2006, 10:45
Post #14





Group: Developer
Posts: 398
Joined: 14-October 01
Member No.: 289



Here, I have uploaded all the files to zshare:

abc-hr.zip - 1.07MB
sample1.zip - 4.36MB
sample2.zip - 2.38MB
sample3.zip - 4.06MB
sample4.zip - 4.12MB
sample5.zip - 2.11MB
sample6.zip - 4.56MB
sample7.zip - 2.81MB
sample8.zip - 5.10MB
sample9.zip - 4.49MB
sample10.zip - 5.06MB
sample11.zip - 5.59MB
sample12.zip - 1.66MB
sample13.zip - 3.07MB
sample14.zip - 1.90MB
Go to the top of the page
+Quote Post
muaddib
post Dec 22 2006, 11:02
Post #15





Group: Developer
Posts: 398
Joined: 14-October 01
Member No.: 289



For most of the tests it is hard for me to determine what is an encoded sample and what is the original.
What do you people think: should results where only low anchor is determined also be sent and counted as relevant data?
Go to the top of the page
+Quote Post
menno
post Dec 22 2006, 13:03
Post #16


Nero MPEG4 developer


Group: Developer (Donating)
Posts: 1218
Joined: 11-October 01
From: LA
Member No.: 267



QUOTE (spockep @ Dec 21 2006, 19:37) *
So in short you just need the .erf files zipped then?

Yeah just zip them and send them to me.
Go to the top of the page
+Quote Post
IgorC
post Dec 23 2006, 03:39
Post #17





Group: Members
Posts: 1573
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



QUOTE (muaddib @ Dec 22 2006, 02:02) *
For most of the tests it is hard for me to determine what is an encoded sample and what is the original.
What do you people think: should results where only low anchor is determined also be sent and counted as relevant data?

That would a half of problem. On some samples after to be enable to abxed them (even high anchor) it's still hard to judge wich one was slightly better than another one.
Go to the top of the page
+Quote Post
muaddib
post Dec 23 2006, 20:47
Post #18





Group: Developer
Posts: 398
Joined: 14-October 01
Member No.: 289



QUOTE (IgorC @ Dec 23 2006, 03:39) *
QUOTE (muaddib @ Dec 22 2006, 02:02) *

For most of the tests it is hard for me to determine what is an encoded sample and what is the original.
What do you people think: should results where only low anchor is determined also be sent and counted as relevant data?

That would a half of problem. On some samples after to be enable to abxed them (even high anchor) it's still hard to judge wich one was slightly better than another one.


Yes, that is a big problem. That is especially evident when average rating for a sample is between 2 and 3. But that problem IMO does not have a solution.

p.s. All your bases are belong to us! wink.gif
Go to the top of the page
+Quote Post
SirChristof
post Jan 5 2007, 09:07
Post #19





Group: Members
Posts: 41
Joined: 2-November 06
Member No.: 37074



My main problem is, in order to properly prepare myself for the "Official" ABC/HR test, I always do very very many ABX tests---and since there are so many samples being tested (7 in this case), by the time I have finished training myself on them via ABX, my ears are too fatigued to take the "official" test in an ideal fashon.

Part of the issue (at least for me) is that some artifacts in some of the samples, are nearly inaudible unless listened to a bit on the loud side. So I get fatigued rather quickly.

I do have solid gear and a very quiet listening environment, so I will make my best attempt to contribute regardless. wink.gif
Go to the top of the page
+Quote Post
menno
post Jan 5 2007, 11:43
Post #20


Nero MPEG4 developer


Group: Developer (Donating)
Posts: 1218
Joined: 11-October 01
From: LA
Member No.: 267



First of all, thanks to everyone who has already contributed!

For the people who still want to participate, please remember that the test runs until the 8th, which is next monday!
Go to the top of the page
+Quote Post
IgorC
post Jan 6 2007, 20:51
Post #21





Group: Members
Posts: 1573
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



menno
Can you confirm if you received my results? Often the messages end in spam folder. Thank you.

This post has been edited by IgorC: Jan 6 2007, 20:51
Go to the top of the page
+Quote Post
menno
post Jan 7 2007, 11:16
Post #22


Nero MPEG4 developer


Group: Developer (Donating)
Posts: 1218
Joined: 11-October 01
From: LA
Member No.: 267



QUOTE (IgorC @ Jan 6 2007, 20:51) *
menno
Can you confirm if you received my results? Often the messages end in spam folder. Thank you.

Yes got them smile.gif thanks.
Go to the top of the page
+Quote Post
muaddib
post Jan 8 2007, 10:49
Post #23





Group: Developer
Posts: 398
Joined: 14-October 01
Member No.: 289



QUOTE (SirChristof @ Jan 5 2007, 09:07) *
My main problem is, in order to properly prepare myself for the "Official" ABC/HR test, I always do very very many ABX tests---and since there are so many samples being tested (7 in this case), by the time I have finished training myself on them via ABX, my ears are too fatigued to take the "official" test in an ideal fashon.

Part of the issue (at least for me) is that some artifacts in some of the samples, are nearly inaudible unless listened to a bit on the loud side. So I get fatigued rather quickly.


IMO it is not important to do so much detailed ABX test.
In tests conducted by ITU (for development of PEAQ) testers were asked to give scores even if they are not certain what is the original and what is a test sample. It was even desired that they don't give 5 but rather something between 4 and 5. So people sometimes judged incorrectly what is the original and what is a test. There are results in ITU tests with positive SDG (meaning that a test sample sounds better than the original).
IMO it is not good to have this kind of results, but it is also better to have results with 5 than not to have any results at all. 5 is also good because it means that you can not hear the difference. So if it is hard for you to ABX simple give up and give 5 smile.gif
Go to the top of the page
+Quote Post
menno
post Jan 8 2007, 13:35
Post #24


Nero MPEG4 developer


Group: Developer (Donating)
Posts: 1218
Joined: 11-October 01
From: LA
Member No.: 267



Because we didn't get so many results yet, we decided to let the test run for one week longer. So now you can send in your results until the 15th of January.
Thanks for your help.

This post has been edited by menno: Jan 8 2007, 13:37
Go to the top of the page
+Quote Post
SirChristof
post Jan 8 2007, 14:07
Post #25





Group: Members
Posts: 41
Joined: 2-November 06
Member No.: 37074



QUOTE (muaddib @ Jan 8 2007, 04:49) *
IMO it is not important to do so much detailed ABX test.
In tests conducted by ITU (for development of PEAQ) testers were asked to give scores even if they are not certain what is the original and what is a test sample. It was even desired that they don't give 5 but rather something between 4 and 5. So people sometimes judged incorrectly what is the original and what is a test. There are results in ITU tests with positive SDG (meaning that a test sample sounds better than the original).
IMO it is not good to have this kind of results, but it is also better to have results with 5 than not to have any results at all. 5 is also good because it means that you can not hear the difference. So if it is hard for you to ABX simple give up and give 5 smile.gif


Determining that I hear a difference is easy. Say we have original, and 2 encoded "test files", O, A and B respectively. Even in such a case where I can hear a difference between O & A, O & B, and A & B, it can be difficult to truly give a "better" answer. Doing very many ABX tests between them (assuming you hear a difference), better allows you to "rate" them. Without such attention to detail, the test becomes less meaningfull, at least in my opinion.

I am still in the process of doing some of these test samples for this listening test.

A small handful of the given test samples, were transparent to me upon my initial listen.

If I did not do many ABX tests to first prepare myself to get accustomed to what I should be hearing, I could simply say 5/5 and move on. When people don't give a full effort, and just listen a couple times then determine "It is transparent", that severely waters down the results from people like me who actually did everything in their power to not only hear the difference, but rate it as honestly as they can.

On a side-note, I did not hear about this test until the 5th, so the extension from the 8th to 15th is welcomed. Thanks menno.
Go to the top of the page
+Quote Post

3 Pages V   1 2 3 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 18th September 2014 - 01:40