IPB

Welcome Guest ( Log In | Register )

8 Pages V  « < 4 5 6 7 8 >  
Closed TopicStart new topic
Public, Multiformat Listening Test @ 64 kbps - OPEN
vinnie97
post Aug 12 2007, 06:39
Post #126





Group: Members
Posts: 472
Joined: 6-March 03
Member No.: 5360



Sebastian, can you tell me if I rated any references? Results should've arrived from vince dot hart at gmail. I fear the answer.
Go to the top of the page
+Quote Post
Sebastian Mares
post Aug 12 2007, 10:16
Post #127





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



The key will be available after the test is over so you can check yourself.

People, about two hours left! wink.gif


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
Sebastian Mares
post Aug 12 2007, 13:05
Post #128





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Hmm... I could use a bit more results for the last 10 samples. sad.gif I am extending the test one last time until 15th since I expected a bit more results to arrive shortly before the test ends. Maybe Guru can test, too (BTW, received two mails from you - thanks!).


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
naylor83
post Aug 12 2007, 17:51
Post #129





Group: Members
Posts: 210
Joined: 19-June 05
From: Uppsala, Sweden
Member No.: 22842



Come on people! If you feel geeky doing the tests, remember it's all in the name of science!


--------------------
davidnaylor.org
Go to the top of the page
+Quote Post
Alex B
post Aug 12 2007, 18:36
Post #130





Group: Members
Posts: 1303
Joined: 14-September 05
From: Helsinki, Finland
Member No.: 24472



Also, even if you are not personally interested about this bitrate range this a good opportunity to practice with relatively easy samples. A future higher bitrate test will be much more difficult for the testers. We have recently tested 48 kbps and now 64 kbps. Possibly the next multiformat test will be 96 kbps. I also look forward to see the discussed 128 kbps MP3 test happening sometime soon.


--------------------
http://listening-tests.freetzi.com
Go to the top of the page
+Quote Post
Whelkman
post Aug 12 2007, 18:45
Post #131





Group: Members
Posts: 91
Joined: 22-January 02
Member No.: 1111



QUOTE (Alex B @ Aug 12 2007, 13:36) *
Also, even if you are not personally interested about this bitrate range this a good opportunity to practice with relatively easy samples. A future higher bitrate test will be much more difficult for the testers. We have recently tested 48 kbps and now 64 kbps. Possibly the next multiformat test will be 96 kbps. I also look forward to see the discussed 128 kbps MP3 test happening sometime soon.

This is why I joined. 48 kbps is unrealistically low and thus uninteresting to me right now, but I'm having problems as early as 96 kbps. A new 128 kbps test may be more challenging than many are expecting. 64 kbps is just the right spot where you might be fooled some of the time, making it interesting for testing purposes.
Go to the top of the page
+Quote Post
Sebastian Mares
post Aug 12 2007, 19:00
Post #132





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Well, in case anyone is interested, the next tests will be:

MP3 at 128 kbps
AAC at 80 kbps (featuring both LC and HE-AAC)
Multiformat at 80 kbps with winner of the AAC test vs. other codecs
And finally maybe a 32 kbps and a 96 kbps test, but these have very low priority

This post has been edited by Sebastian Mares: Aug 12 2007, 19:01


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
Alex B
post Aug 12 2007, 21:15
Post #133





Group: Members
Posts: 1303
Joined: 14-September 05
From: Helsinki, Finland
Member No.: 24472



QUOTE (Sebastian Mares @ Aug 12 2007, 21:00) *
Well, in case anyone is interested, the next tests will be:

MP3 at 128 kbps
AAC at 80 kbps (featuring both LC and HE-AAC)
Multiformat at 80 kbps with winner of the AAC test vs. other codecs
And finally maybe a 32 kbps and a 96 kbps test, but these have very low priority

It is good to see that you have plans for several tests since no one else has been interested about conducting listening tests.

The 128 kbps MP3 test will be interesting because LAME 3.98 should be soon ready and FhG has a new version too.

Personally, I think 80 kbps is quite near of 64 kbps and 96 kbps would be a better next step for a multiformat test, but I suppose you have carefully thought about the options and followed the ongoing HA discussions more keenly than I.

It would be good to keep the tests flowing in a steady stream and not allow as long breaks as we had now. I think more testers would be willing to participate if the test discussions are constantly kept alive. For example, if a test is not possible in the scheduled time frame because of codec development reasons you could start the next test from the list instead.

Also, I wonder if the test presentations, instructions/help/faq and the results presentations could be developed further. It would be nice to have an attractive and carefully organized web site where you could browse info about the ongoing test, the test schedule and the results easily. Perhaps naylor83 would be interested in helping with this. He did excellent work with the rarewares site.


--------------------
http://listening-tests.freetzi.com
Go to the top of the page
+Quote Post
kennedyb4
post Aug 12 2007, 21:36
Post #134





Group: Members
Posts: 772
Joined: 3-October 01
Member No.: 180



I too would be more interested in a multi-format at 96 with the AAC candidates being Nero and Itunes VBR.

This bitrate would be of more interest for those using portable devices I think.
Go to the top of the page
+Quote Post
IgorC
post Aug 13 2007, 01:46
Post #135





Group: Members
Posts: 1579
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



QUOTE (naylor83 @ Aug 12 2007, 08:51) *
Come on people! If you feel geeky doing the tests, remember it's all in the name of science!

And science should be in the name of better life laugh.gif
Go to the top of the page
+Quote Post
vinnie97
post Aug 13 2007, 05:49
Post #136





Group: Members
Posts: 472
Joined: 6-March 03
Member No.: 5360



QUOTE (Alex B @ Aug 12 2007, 12:15) *
Personally, I think 80 kbps is quite near of 64 kbps and 96 kbps would be a better next step for a multiformat test, but I suppose you have carefully thought about the options and followed the ongoing HA discussions more keenly than I.

I disagree. 48 --> 64 --> 80 = 16 kbps between each test iteration. The difference between Vorbis at 64 and 80 kbps has me very interested in this test. 80 kbps is the portable sweet spot for me when it comes to Vorby...I'd like to see if it's still #1 or if there's a similar transparency being reached across all formats.

Sebastian, you're killing me....I need to know how many references I ranked. laugh.gif

This post has been edited by vinnie97: Aug 13 2007, 05:50
Go to the top of the page
+Quote Post
rockcake
post Aug 13 2007, 06:25
Post #137





Group: Members (Donating)
Posts: 26
Joined: 22-December 05
Member No.: 26569



I'm also with vinnie97 wholeheartedly as regards 80 kbps tests - I'd also be interested in the results & suspect quite a few participants would be pleasantly surprised with the fidelity of the sounds; I for one was a little surprised in this test at the fidelity in some of the non-problem/general-music samples e.g. "Big Yellow" & "Les Voleurs" come to mind as being mostly quite listenable to my ears/head/brain IIRC.
Can't wait for the overall results!

Edit P.S.: I also dread my contributions being thrown out for whatever reason, so I tend to be a bit conservative and listen/rank like beto wrote above.
P.P.S.: Blimey, the next 128 mp3 test will be ruddy hard, I reckon!

This post has been edited by rockcake: Aug 13 2007, 06:35


--------------------
Vorbis aoTuV b5 -q2 is fine for me.
Go to the top of the page
+Quote Post
muaddib
post Aug 13 2007, 10:33
Post #138





Group: Developer
Posts: 401
Joined: 14-October 01
Member No.: 289



QUOTE (Sebastian Mares @ Aug 12 2007, 14:05) *
Hmm... I could use a bit more results for the last 10 samples. sad.gif I am extending the test one last time until 15th since I expected a bit more results to arrive shortly before the test ends. Maybe Guru can test, too (BTW, received two mails from you - thanks!).

It would be really great if people with golden ears would help us with this test. I guess for them it would be very easy and they could do it in 2-3 hours.
Go to the top of the page
+Quote Post
kdo
post Aug 13 2007, 10:57
Post #139





Group: Members (Donating)
Posts: 304
Joined: 18-April 02
From: Russia
Member No.: 1812



QUOTE (muaddib @ Aug 13 2007, 11:33) *
people with golden ears would help us with this test

But the golden ears would have to rate all these samples between annoying and very annoying.

Whereas people with average hearing can use the full scale for the ratings. Like I did.


QUOTE (Sebastian Mares @ Aug 12 2007, 14:05) *
Hmm... I could use a bit more results for the last 10 samples.


I have a question: When somebody ranks the reference, do you disqualify the whole test for this sample or only the result for this particular encoder within the test?

Would it make any sense for those of us who ranked the references to repeat the discarded tests more carefully and re-submit?
Go to the top of the page
+Quote Post
muaddib
post Aug 13 2007, 11:07
Post #140





Group: Developer
Posts: 401
Joined: 14-October 01
Member No.: 289



QUOTE (kdo @ Aug 13 2007, 11:57) *
QUOTE (muaddib @ Aug 13 2007, 11:33) *
people with golden ears would help us with this test
But the golden ears would have to rate all these samples between annoying and very annoying.
Whereas people with average hearing can use the full scale for the ratings. Like I did.

Giving low grades wouldn't make any problems.
And because someone can hear very well and find those small differences it doesn't mean that he/she will grade those differences as annoying.
Go to the top of the page
+Quote Post
guruboolez
post Aug 13 2007, 12:12
Post #141





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (muaddib @ Aug 13 2007, 10:33) *
It would be really great if people with golden ears would help us with this test. I guess for them it would be very easy and they could do it in 2-3 hours.

I'll do it today or maybe tomorrow for all 18 samples - Sebastian will therefore have one more result for each one.
Go to the top of the page
+Quote Post
Alex B
post Aug 13 2007, 12:46
Post #142





Group: Members
Posts: 1303
Joined: 14-September 05
From: Helsinki, Finland
Member No.: 24472



I think the amount of ranked references in the future tests could be lessened with better instructions. ff123's practise page is very useful, but an updated page (with the JAVA version screenshots) inside the test site would be better.


The following things should be said very clearly:

- If the difference is not obvious, you should do an ABX test.
- If you can't succesfully ABX you should not try to guess. Don't pull either of the rating sliders down.
- If you didn't ABX test an obvious sample it would be good to listen to it once again and verify that you moved the correct slider.


Perhaps the instructions could have some step by step examples with screenshots. For example:

[Screenshot 1, the initial view] & a speech bubble:
"Hmmm... the sample on the left has less overall clarity and the symbals are a bit diminished, could it be the lossy file? Though, I am not sure, better go to ABX."

[Screenshot 2, ABX view, configuring the test]

[Screenshot 3, ABX view after a succesful ABX test] & a speech bubble:
"Oh dear, it was the other way around. This encoder has increased the volume of the higher frequencies slightly and at first sight it sounded better because the reference is not very bright."

and so on...


--------------------
http://listening-tests.freetzi.com
Go to the top of the page
+Quote Post
naylor83
post Aug 13 2007, 13:08
Post #143





Group: Members
Posts: 210
Joined: 19-June 05
From: Uppsala, Sweden
Member No.: 22842



QUOTE (Alex B @ Aug 13 2007, 13:46) *
"Oh dear, it was the other way around. This encoder has increased the volume of the higher frequencies slightly and at first sight it sounded better because the reference is not very bright."


"... Ah, that must mean this is the WMA sample! I'll bring it down an extra notch just for spite."

biggrin.gif

Edit: Seriously though... Good suggestions!

This post has been edited by naylor83: Aug 13 2007, 13:09


--------------------
davidnaylor.org
Go to the top of the page
+Quote Post
muaddib
post Aug 13 2007, 13:21
Post #144





Group: Developer
Posts: 401
Joined: 14-October 01
Member No.: 289



All participants in this test that have saved their sessions, could use Alex B instructions to verify that no references were ranked.

I am willing to help in building instructions for future listening tests (for this one it is to late I am afraid). Maybe someone could start this in Hydrogenaudio Wiki (if that is a good place).
Go to the top of the page
+Quote Post
kennedyb4
post Aug 13 2007, 15:02
Post #145





Group: Members
Posts: 772
Joined: 3-October 01
Member No.: 180



QUOTE (muaddib @ Aug 13 2007, 08:21) *
All participants in this test that have saved their sessions, could use Alex B instructions to verify that no references were ranked.

I am willing to help in building instructions for future listening tests (for this one it is to late I am afraid). Maybe someone could start this in Hydrogenaudio Wiki (if that is a good place).


That's a fantastic idea. I downloaded the program and a few samples but got bogged down in the mechanics of things. Wasn't really sure what I was doing.

A single clear " how to " manual might have helped.
Go to the top of the page
+Quote Post
kdo
post Aug 13 2007, 15:02
Post #146





Group: Members (Donating)
Posts: 304
Joined: 18-April 02
From: Russia
Member No.: 1812



QUOTE (muaddib @ Aug 13 2007, 14:21) *
All participants in this test that have saved their sessions, could use Alex B instructions to verify that no references were ranked.

In my case it would mean practically re-doing the whole test from scratcg, because most of the time I didn't do abx.
I did go through abx a few times when I was in doubt, and I did manage to abx successfully every time (or almost every, cannot remember). But there is still a chance I could have ranked a reference somtimes.

It would be nice if Sebastian could notify us by e-mail which of our tests are discarded and which we should re-do and re-submit. (Of course, if such thing is allowed by the test protocol)
Go to the top of the page
+Quote Post
guruboolez
post Aug 13 2007, 15:39
Post #147





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



The four first samples are tested on my side.
In order to avoid some mistakes I begin the procedure by ABXing all files and I start ranking them after this ABX procedure. Advantage: it's impossible to accidentaly mark the reference (only one slider can be lowered after a successful ABX test).

N.B. It seems that one competitor tends to make higher frequencies brighter (more noise/energy) which wouldn't be unpleasant if there wasn't additional artefacts. After lossy encoders, maybe the first "glossy" codec... wink.gif

This post has been edited by guruboolez: Aug 13 2007, 15:40
Go to the top of the page
+Quote Post
muaddib
post Aug 13 2007, 15:53
Post #148





Group: Developer
Posts: 401
Joined: 14-October 01
Member No.: 289



QUOTE (guruboolez @ Aug 13 2007, 16:39) *
The four first samples are tested on my side.

I hope you will not get bored until you get to samples 7.-18. smile.gif

I don't understand why so little results for sample 8. It is easy after short 'getting used to it'. However I will not send my results because we at Nero don't want to have influence on the results.
Go to the top of the page
+Quote Post
Whelkman
post Aug 13 2007, 16:09
Post #149





Group: Members
Posts: 91
Joined: 22-January 02
Member No.: 1111



Add to the list explanation of ABX testing behavior. It was far from obvious to me, and I had to redo several samples after experimentation (at the time I thought using the testing mode to eliminate the reference was required in all cases, but apparently this isn't true--this should be covered, too).

Not obvious is that the listener gets only one shot at testing mode unless he wishes to redo all samples within the test or load from a good session. The program also has some strange ideas of what qualifies as a commit; if I begin testing, reset my trials, then decide to come back later, I'll find the test was marked as a failed commit and will not be able to try again without starting over or loading from a successful session save. At least twice I successfully ABXed only to have my results discarded and replaced with a failed commit.

The listener should also know that he needs to run at least eight trials in testing mode. Early on I tried finalizing when ABC/HR first displayed a number (6/6, I believe), but results came up in red and the reference was not discarded, which is not helpful, especially given the retesting issue.

I experienced numerous bugs while using ABC/HR Java including crashes, disappearing menu options, nonfunctional sliders, failed saves, and phantom ABX commits. The program becomes especially unstable after a session save. Are these issues known?

Had I not kept notes on each sample, I probably would have quit due to the above frustrations.
Go to the top of the page
+Quote Post
[JAZ]
post Aug 13 2007, 16:51
Post #150





Group: Members
Posts: 1797
Joined: 24-June 02
From: Catalunya(Spain)
Member No.: 2383



QUOTE (Whelkman @ Aug 13 2007, 17:09) *
I experienced numerous bugs while using ABC/HR Java including crashes, disappearing menu options, nonfunctional sliders, failed saves, and phantom ABX commits. The program becomes especially unstable after a session save. Are these issues known?

Had I not kept notes on each sample, I probably would have quit due to the above frustrations.


You play hard, it seems... wink.gif

I've done three listening tests so far, and the only bugs i found were the non working audio (that gets fixed manually selecting the output audiocard), and the clicks when pressing play (which doesn't happen to me if i use the soundcard directly instead of java sound engine)

But if you say it happens *After* saving session... i've never saved session. (don't use to get interrupted in the middle of a test)

This post has been edited by [JAZ]: Aug 13 2007, 16:55
Go to the top of the page
+Quote Post

8 Pages V  « < 4 5 6 7 8 >
Closed TopicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 29th November 2014 - 06:17