IPB

Welcome Guest ( Log In | Register )

3 Pages V  < 1 2 3 >  
Closed TopicStart new topic
Public Multiformat Listening Test @ ~64 kbps [March/April 2011], Nero & Apple HE-AAC, aoTuV Vorbis, CELT
C.R.Helmrich
post Mar 23 2011, 21:26
Post #26





Group: Developer
Posts: 686
Joined: 6-December 08
From: Erlangen Germany
Member No.: 64012



QUOTE (Jillian @ Mar 21 2011, 20:02) *
I want to ask everyone. Before the test finish, which codec will win? Of course you may have to take a guess/bet or based on your experience/personal preference.

I have certain vague expectations about which codec will win, but I won't tell you (and I ask everyone else with knowledge about the codecs under test not to do so either) since this could influence the test.

A (IMHO) more important question: Igor, are all session results excluded for which some hidden reference is graded with less than 5.0? Maybe you should mention the post-screening procedure so that listeners behave carefully enough.

And about the resampling: Except for CELT, was the input or the output of the codecs resampled to 48 kHz?

Chris

This post has been edited by C.R.Helmrich: Mar 23 2011, 21:29


--------------------
If I don't reply to your reply, it means I agree with you.
Go to the top of the page
+Quote Post
IgorC
post Mar 23 2011, 21:41
Post #27





Group: Members
Posts: 1572
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Hello, Chris.

QUOTE (C.R.Helmrich @ Mar 23 2011, 17:26) *
A (IMHO) more important question: Igor, are all session results excluded for which some hidden reference is graded with less than 5.0? Maybe you should mention the post-screening procedure so that listeners behave carefully enough.

You're right.
Here are the rules:
CODE
If at least one of the following errors takes place then the result will be INVALID.
1. graded the reference lower than 5.0
2. didn't grade the low anchor.

If one listener will submit INVALID result then he will be informed and will have one more and unique posibility to submit result for one particular sample.
Now if the listener will submit 3 or more invalid results then only ABX results will be accepted from him/her or directly rejected in abussive cases.


QUOTE (C.R.Helmrich @ Mar 23 2011, 17:26) *
And about the resampling: Except for CELT, was the input or the output of the codecs resampled to 48 kHz?

Chris

The output of decoded (vorbis, aac) files was resampled.

This post has been edited by IgorC: Mar 23 2011, 21:45
Go to the top of the page
+Quote Post
Fool_on_the_hill
post Mar 24 2011, 11:31
Post #28





Group: Members
Posts: 88
Joined: 30-October 05
From: Russia, Tomsk
Member No.: 25459



Tried to abchr.jar (+jdom.jar, looks-1.3.1.jar) from here http://www.hydrogenaudio.org/forums/index....st&p=701994?
The same result. Strange...
Go to the top of the page
+Quote Post
LithosZA
post Mar 24 2011, 14:17
Post #29





Group: Members
Posts: 196
Joined: 26-February 11
Member No.: 88525



QUOTE
Tried to abchr.jar (+jdom.jar, looks-1.3.1.jar) from here http://www.hydrogenaudio.org/forums/index....st&p=701994?
The same result. Strange...


You could try uninstalling the 64bit version of JRE/JDK and try using the 32bit version of the java runtime instead.
Don't think it will make a difference, but it is worth a try.
Go to the top of the page
+Quote Post
Sebastian Mares
post Mar 24 2011, 16:26
Post #30





Group: Members
Posts: 3629
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Your JRE is probably too new. smile.gif Had this problem in my tests, too.


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
IgorC
post Mar 24 2011, 16:48
Post #31





Group: Members
Posts: 1572
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Thank you, Sebastian.

There are two versions of ABC-HR for Java 5 and 6. http://www.hydrogenaudio.org/forums/index....st&p=683924

Fool_on_the_hill, you can try it

CODE
The Java 5 build should also run fine on Java 6 machines. In both cases just double-click the jar to start.


This post has been edited by IgorC: Mar 24 2011, 16:56
Go to the top of the page
+Quote Post
C.R.Helmrich
post Mar 24 2011, 22:34
Post #32





Group: Developer
Posts: 686
Joined: 6-December 08
From: Erlangen Germany
Member No.: 64012



Some relevant knowledge collected for the 128-kb test that didn't take place (yet...):

Instruction sheet for audio codec listening tests (high quality)

(German: eine deutsche Version der Anleitung liegt hier).

The best codecs in this 64-kb test can still be considered high-quality, so the description is applicable.

By the way, two posts above mine in that thread is a Mac binary of faad. In case any Mac user needs it. [Edit: unfortunately, not any more sad.gif ]

Chris

This post has been edited by C.R.Helmrich: Mar 24 2011, 22:46


--------------------
If I don't reply to your reply, it means I agree with you.
Go to the top of the page
+Quote Post
Alex B
post Mar 25 2011, 10:17
Post #33





Group: Members
Posts: 1303
Joined: 14-September 05
From: Helsinki, Finland
Member No.: 24472



Thanks for organizing this test.


To help the testers who do not have access to a Windows computer I created FLAC versions of the decoded samples and put them online on Dropbox.

I decided to pack the sample folders into six zip files (a compromise between storing them individually or in a single package). Each zip file contains five sample folders:

http://dl.dropbox.com/u/7690288/samples01-05.zip (48.6 MB)
http://dl.dropbox.com/u/7690288/samples06-10.zip (67.3 MB)
http://dl.dropbox.com/u/7690288/samples11-15.zip (67.6 MB)
http://dl.dropbox.com/u/7690288/samples16-20.zip (52.7 MB)
http://dl.dropbox.com/u/7690288/samples21-25.zip (61.6 MB)
http://dl.dropbox.com/u/7690288/samples25-30.zip (67.7 MB)

I compressed the original, decoded 48 kHz WAVE test samples with FLAC 1.2.1 (-8) and verified the resulting FLAC files with foobar2000's bit compare tool.

The sample folders are included in the zip files. You must unzip the folders and decompress the FLAC files back to WAVE before they can be used in the test. The new filename extension must be .wav instead of .flac, otherwise the filenames must be unaltered. The folders must be placed in the ABC-HR folder:

ABC-HR > Sample01 > the six uncompressed .wav sample files
ABC-HR > Sample02 > the six uncompressed .wav sample files
ABC-HR > Sample03 > the six uncompressed .wav sample files
etc.

The 30 sample folders (= 180 WAVE files) will need about 712 MB of disk space.

You can have the FLAC files in the same folders if you don't want to delete them. They will just occupy some additional disk space (about 366 MB for all 30x6 samples).


Please don't abuse the bandwidth that is available for this Dropbox account. It makes no sense to download these versions if you can use the smaller original package (or packages) and let the bat file (or files) automatically process the test samples.

EDIT

If anyone else wants to host these FLAC versions I can provide also the individual sample packages and all samples in a single package (PM me).

This post has been edited by Alex B: Mar 25 2011, 10:46


--------------------
http://listening-tests.freetzi.com
Go to the top of the page
+Quote Post
IgorC
post Mar 25 2011, 11:08
Post #34





Group: Members
Posts: 1572
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Thank you, AlexB
That is exactly what people asking with Linux and OS X.
Go to the top of the page
+Quote Post
IgorC
post Mar 25 2011, 17:25
Post #35





Group: Members
Posts: 1572
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Here is more clear interpretation of rules:

QUOTE
Participants who don't want to worry too much about the grading rules
can simply ignore them. Listeners should do their best to rank the samples and
be careful to identify the hidden references. Listeners should ABX tests
they are at all unsure.

1) If the low anchor is not graded, or if any hidden reference is graded
below 4.5-5(*1a) (see App.) the result is INVALID.

*1a It will be discussed with people who have experience of conducting public tests when final results will be ready.

2) For each sample with a ranked reference or an ungraded low anchor the
listener will have a single chance to submit a replacement test run for
that sample. The replacement test must cover all codecs, not just the
codecs with the ranked reference. (This also covers cases where the
reference is ranked but still at or above 4.5)

3) If a listener submits more than 2/15 (4 for 30 samples submitted) INVALID
results then only ABX results will be accepted, or the listener will be excluded
completely in cases of apparently abusive behavior.


App. These rules aren't extremely strict in order to allow for simple human
error while still excluding careless participants.

A stricter procedure to exclude all ranked references risks a systemic
bias against any codec which are very good on a few samples and thus
subject to more reference confusion by causing those samples to be excluded
and weighing the test towards other samples.
Go to the top of the page
+Quote Post
Alex B
post Mar 25 2011, 17:39
Post #36





Group: Members
Posts: 1303
Joined: 14-September 05
From: Helsinki, Finland
Member No.: 24472



I just noticed that this test is not in the forum news.


--------------------
http://listening-tests.freetzi.com
Go to the top of the page
+Quote Post
IgorC
post Mar 26 2011, 02:26
Post #37





Group: Members
Posts: 1572
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



QUOTE (Alex B @ Mar 25 2011, 13:39) *
I just noticed that this test is not in the forum news.

Oh, it will be great to have it there

This post has been edited by IgorC: Mar 26 2011, 02:26
Go to the top of the page
+Quote Post
Canar
post Mar 26 2011, 03:37
Post #38





Group: Super Moderator
Posts: 3360
Joined: 26-July 02
From: princegeorge.ca
Member No.: 2796



Create a post in the News Submissions forum and I'll see what I can do. smile.gif


--------------------
You cannot ABX the rustling of jimmies.
No mouse? No problem.
Go to the top of the page
+Quote Post
Alex B
post Mar 26 2011, 10:24
Post #39





Group: Members
Posts: 1303
Joined: 14-September 05
From: Helsinki, Finland
Member No.: 24472



Actually the first post in this thread is the news announcement.

During the past tests we usually had 1) a pretest discussion in "Listening Tests", 2) a news thread in which the open test was announced and commented and finally 3) a news thread in which the results were published and commented.

For example, the previous Multiformat 64 kbps test:

1) pretest discussion: http://www.hydrogenaudio.org/forums/index....showtopic=53134
2) announcement: http://www.hydrogenaudio.org/forums/index....showtopic=56397
3) results: http://www.hydrogenaudio.org/forums/index....showtopic=56851

This time we did not have a pretest discussion (at least not in a single dedicated thread). This thread is already the news & comments thread.


To: Admins/Mods

- Could you simply create a short (perhaps slightly edited) news article from the first post that would point to this thread?
or
- Could you move this thread to "Validated News", create a short (perhaps slightly edited) news article from the first post that would point to this thread and perhaps leave here, in "Listening Tests", a short note about the moved thread and a link to the new location?


--------------------
http://listening-tests.freetzi.com
Go to the top of the page
+Quote Post
IgorC
post Mar 26 2011, 16:29
Post #40





Group: Members
Posts: 1572
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



AlexB has a point. We need the announcement of this test on News Section. This way more members will participate.

Some members have already sent their results.
it's on approximately half way to get enough results.
Hopefully people will have time to conduct test during this weekend.

Hydrogenaudio community, please, participate in test! biggrin.gif

This post has been edited by IgorC: Mar 26 2011, 16:34
Go to the top of the page
+Quote Post
IgorC
post Mar 27 2011, 17:18
Post #41





Group: Members
Posts: 1572
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



BTW, If you don't want to download separate packages there is All-In-One package (ABC-HR and all samples)
It should be more easy.

ABC-HR_bin_and_samples (All-In-One)

This post has been edited by IgorC: Mar 27 2011, 17:21
Go to the top of the page
+Quote Post
LithosZA
post Mar 27 2011, 21:31
Post #42





Group: Members
Posts: 196
Joined: 26-February 11
Member No.: 88525



The test was really fun. I was supprised how nearly transparent some of these samples were at 64Kbit/s.
I would encourage that more people take this test.
Go to the top of the page
+Quote Post
Fool_on_the_hill
post Mar 28 2011, 13:24
Post #43





Group: Members
Posts: 88
Joined: 30-October 05
From: Russia, Tomsk
Member No.: 25459



QUOTE (IgorC @ Mar 24 2011, 09:48) *
Thank you, Sebastian.

There are two versions of ABC-HR for Java 5 and 6. http://www.hydrogenaudio.org/forums/index....st&p=683924

Fool_on_the_hill, you can try it

CODE
The Java 5 build should also run fine on Java 6 machines. In both cases just double-click the jar to start.


still the same error
Go to the top of the page
+Quote Post
IgorC
post Mar 28 2011, 18:46
Post #44





Group: Members
Posts: 1572
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



QUOTE (Fool_on_the_hill @ Mar 28 2011, 09:24) *
still the same error


It's a sort of particular incompatibility. I have the same Windows 7 x64 and Java 5/6 packages work fine on it. Also have checked on Window XP machine. No problem. mellow.gif

This post has been edited by IgorC: Mar 28 2011, 19:01
Go to the top of the page
+Quote Post
IgorC
post Mar 29 2011, 19:24
Post #45





Group: Members
Posts: 1572
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Some people wait for the last day to send their results. That's ok. laugh.gif
But it might be the case that someone can forget to send the results until 30th of March.
So if You have already even a few results send them soon.

Of course, the last moment results will be accepted during 30th and 31st (~01.00 of GMT -3.00) of March.

As far as I can see all participants have received the answer. Thank You to All who participate.



This post has been edited by IgorC: Mar 29 2011, 20:23
Go to the top of the page
+Quote Post
Sebastian Mares
post Mar 29 2011, 19:41
Post #46





Group: Members
Posts: 3629
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



I usually got a decent number of results on the last day from people who wanted to test more but didn't make it in time. wink.gif


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
markanini
post Mar 29 2011, 22:56
Post #47





Group: Members
Posts: 550
Joined: 22-December 03
From: Malmö, Sweden
Member No.: 10615



I've had fun doing the tests but please make the samples shorter next time so that the participant can more easily establish points in the audio to focus on and avoid fatigue. For example sample 15 starts of with a female voice and then switches to a male voice, forces you to acquire a split personality biggrin.gif
Go to the top of the page
+Quote Post
lessthanjoey
post Mar 30 2011, 05:44
Post #48





Group: Members
Posts: 17
Joined: 23-March 11
Member No.: 89226



I'm extremely impressed by all four (non low anchor) codecs. Two in particular appear essentially transparent to my ear across the entire set of samples. I can only hope that those two are consistent and it's not a cycling of which is best in each case smile.gif
Go to the top of the page
+Quote Post
romor
post Mar 30 2011, 06:45
Post #49





Group: Members
Posts: 672
Joined: 16-January 09
Member No.: 65630



QUOTE (lessthanjoey @ Mar 30 2011, 05:44) *
Two in particular appear essentially transparent to my ear across the entire set of samples. I can only hope that those two are consistent and it's not a cycling of which is best in each case smile.gif

Notices the same - I downloaded bundle out of curiosity and 2 codecs perform superbly on couple of samples I tested

2m to midnight - is there minimum amount of samples that can be reported? Like 1/3 of all or 10 for reckless ones? cool.gif

edit: I realised now (readme.txt) that it's not necessary to test all samples, sorry for wasting space

This post has been edited by romor: Mar 30 2011, 06:52


--------------------
scripts: http://goo.gl/M1qVLQ
Go to the top of the page
+Quote Post
IgorC
post Mar 30 2011, 19:40
Post #50





Group: Members
Posts: 1572
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



The test is extended until the 10th of April.
We still need more results for some particular samples.
I will post the numbers of those samples in a few minutes.

This post has been edited by IgorC: Mar 30 2011, 19:40
Go to the top of the page
+Quote Post

3 Pages V  < 1 2 3 >
Closed TopicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 16th September 2014 - 12:25