Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: AAC @ 128kbps listening test discussion (Read 84178 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

AAC @ 128kbps listening test discussion

Reply #225 –
Can't remember their name or number in the list. I've reported my feelings in the ABC/HR comments; I just need the decrypted files to answer you.

P.S. The first one troubles me, and waiting.wav is heavily distorted (or at least very unatural voice) too. But there are some other files too.
It's probably personal troubles, linked to musical genre and mastering I'm not used to listen to (some people are troubled with classical: maybe too much fidelity )

AAC @ 128kbps listening test discussion

Reply #226 –
Quote
I agree. And one of them suffers from excessive lowpass to my ears (~15000 hertz, maybe less). Maybe not as bad as ringing or other distortions in daily playback, but on ABA/ABX comparisons, the consequences of the lowpass are really unpleasant and immediately betray the encoding.

I noticed this also (I think I mentioned it in the results), particularly on one of the samples.

AAC @ 128kbps listening test discussion

Reply #227 –
Results send for the 12 samples.

After some warming up I didn't have much trouble with the test. Most samples were easily picked because of obvious lowpass (my hearing goes up to 19khz) and ringing/smearing. Sample 5 was a bitch though, could't pick any until I found a spot were preecho occurred with one sample. Samples 1, 2, 6 and 10 were hard, too.

gear: Terratec DMX6Fire 24/96, oldschool Toshiba HR-80 Headphones 

Among the encoders are two very good ones and two quite bad.

Looking forward to the results. Will there be an extension?

AAC @ 128kbps listening test discussion

Reply #228 –
Quote
Looking forward to the results. Will there be an extension?

Yes. AAC winner vs. MP3 winner (Lame) vs. WMA Std., MPC, Vorbis and anchor.

AAC @ 128kbps listening test discussion

Reply #229 –
Are they enough results for the moment? I really hope to see the final results in 48 hours, and not in 10 days...

AAC @ 128kbps listening test discussion

Reply #230 –
Quote
Are they enough results for the moment?

Hard to say. I surely received enough results, but I'm getting a big amount of ranked references. Some of them can still be used (because the listener sucessfully ABXd the ranked reference), some must be dropped because no ABX was performed.

I'm still screening them to see what can be used and what can't. Hopefully there will be enough by tomorrow night.

AAC @ 128kbps listening test discussion

Reply #231 –
Quote
Quote
Are they enough results for the moment?

Hard to say. I surely received enough results, but I'm getting a big amount of ranked references. Some of them can still be used (because the listener sucessfully ABXd the ranked reference), some must be dropped because no ABX was performed.

I'm still screening them to see what can be used and what can't. Hopefully there will be enough by tomorrow night.

Can ranked references be considered as transparent?
A ranked reference shows, that the listener "fails" at this sample and other effects (psychological, fitness) are responsible for the ranking.
I isn't this in some way the same as transparent?

AAC @ 128kbps listening test discussion

Reply #232 –
Quote
Can ranked references be considered as transparent?

Yes, they can.

But I can't accept all ranked references, or I would be acepting files where the participant just randomly moved some sliders and sent me the results (like a very famous result set I received in the 64kbps test)

That's why I'm going to use ABX results to decide what will be used and what will be dropped.

AAC @ 128kbps listening test discussion

Reply #233 –
i sent in my results now (i hope i dont have too much ranked references, its hard to avoid with encrypted results  )
I know, that I know nothing (Socrates)

AAC @ 128kbps listening test discussion

Reply #234 –
I'm really impatient to see my results, and the possible mistakes I did 

AAC @ 128kbps listening test discussion

Reply #235 –
Quote
That's why I'm going to use ABX results to decide what will be used and what will be dropped.

What happens in this case:

I successfully abxed (p < 0.01) but afterwards I'm exausted and can't hear the difference anymore reliably and rank the reference - Will this cout as 5.0?
Let's suppose that rain washes out a picnic. Who is feeling negative? The rain? Or YOU? What's causing the negative feeling? The rain or your reaction? - Anthony De Mello

AAC @ 128kbps listening test discussion

Reply #236 –
Quote
I successfully abxed (p < 0.01) but afterwards I'm exausted and can't hear the difference anymore reliably and rank the reference - Will this cout as 5.0?

Yes. I discussed the subject with Garf and ff123 (that are the two other Hydrogenaudio members with experience in public listening tests), and they agree that scoring 5 for ranked references successfully ABXd is OK.

AAC @ 128kbps listening test discussion

Reply #237 –
results sent... phew

AAC @ 128kbps listening test discussion

Reply #238 –
Hmm... strikes me as a little unfair. If someone went through the hassle of ABXing all the way to p < 0.01, a mere slip or lapse of concentration may make all that effort worthless, since s/he might as well ranked it 5.0 right away with the first difficult sample and not ABXed at all. If I were in the situation of having to decide, I would 'switch' the rank of the reference to the sample, provided that the (misplaced) rank is above 4.0 (or maybe even a little higher).
An encrypted ABX result of p < 0.01 is a very strong evidence that the person IS detecting a difference and therefore, you are discarding valuable and valid information by assigning a 5.0 to an obvious mistake in the ranking. In the case of subtle, neutral artifacts (which would be expected to be ranked at >4.0), it would not be too difficult for a fatigued or distracted person to confound the encoding with the original, considering that none sounds annoying to him/her. Ranking a reference as already annoying, however, well...  ... we simply can't take the person's word for it and hence should be taken as 5.0, even in view of a successful ABX.

AAC @ 128kbps listening test discussion

Reply #239 –
The best solution IMHO would be, to reduce the two sliders to one after a significant ABX-result.

AAC @ 128kbps listening test discussion

Reply #240 –
well what if i rank the reference without abxing?

i mean i didnt do abx on all samples or till a result < 0.01 (yes i am lazy) when i thought i hear a difference right away (well who knows if i was right with these)

i mean i would think it would be ok to use these ranked references as 5.0 too (i think most people are reliable here to not send in randomly choosen results, like me    )
I know, that I know nothing (Socrates)

AAC @ 128kbps listening test discussion

Reply #242 –
Quote
I'm conducting the test right now, but sometimes the applet crashes

Well, if you want me to do something about it, you have to be a bit more precise in your error description... Either tell me how to reproduce the crash or start the application from a console and send me the error message. Thanks.

AAC @ 128kbps listening test discussion

Reply #243 –
Quote
Quote
I'm conducting the test right now, but sometimes the applet crashes

Well, if you want me to do something about it, you have to be a bit more precise in your error description... Either tell me how to reproduce the crash or start the application from a console and send me the error message. Thanks.

Crashed here too occasionally. Since I wanted to finish the test without unnecessary repetitions I didn't try to reproduce the problem. I'll try to describe as good as I can though:

- Crash means that it becomes unresponsive, e.g. ABX window can't be closed anymore, when moving another window in foreground, things disappear, only possibility to close it down is Task manager (Win2ksp4 here).

- These crashs happened when I did too much with the mouse, e.g. changing playback range before playback is finished (normally playback just stops, but sometimes crashs happen) or when changing between A,B and X very often during short time (Fast switching disabled here).

I can try to reproduce the problem if you tell what exactly "start the application from a console" means - and what I have to do.


BTW: Results sent. Was a hard piece of work...
Let's suppose that rain washes out a picnic. Who is feeling negative? The rain? Or YOU? What's causing the negative feeling? The rain or your reaction? - Anthony De Mello

AAC @ 128kbps listening test discussion

Reply #244 –
Quote
I can try to reproduce the problem if you tell what exactly "start the application from a console" means - and what I have to do.

By this I mean opening a command prompt and using "java -jar abchr.jar" (in the directory where you unzipped the files) to start the application, instead of just clicking on abchr.jar in Explorer. If there's a fatal crash, Java will print debug information which can be useful to find the error.
Won't be necessary in this case, however. I'm pretty sure what the cause of these effects is. Thanks for your description, I'll try to fix it in the next version.

AAC @ 128kbps listening test discussion

Reply #245 –
It may be the same crash, but well...

ABC-HR Java 0.4b3 SE crashes when I press stop at the end of the file (less than buffer length).
It bugs out most often with buffer length set to 2000ms.

Gentoo Linux post-2004.0, built with 2.6.3 headers and New Posix Thread Library.
Kernel 2.6.3-mm4

Terratec Aureon 7.1 on ALSA from the kernel, OSS emulation on (1.0.2c?)
ALSA-lib/utils - 1.0.2
No esound/arts/anything.

java version "1.4.1"
Java™ 2 Runtime Environment, Standard Edition (build Blackdown-1.4.1-01)
Java HotSpot™ Client VM (build Blackdown-1.4.1-01, mixed mode)

<edit>
More info: No loop, No fast switching

No text printed to std{out,err} on crash.
ruxvilti'a

AAC @ 128kbps listening test discussion

Reply #246 –
Quote
Quote
I can try to reproduce the problem if you tell what exactly "start the application from a console" means - and what I have to do.

By this I mean opening a command prompt and using "java -jar abchr.jar" (in the directory where you unzipped the files) to start the application, instead of just clicking on abchr.jar in Explorer. If there's a fatal crash, Java will print debug information which can be useful to find the error.
Won't be necessary in this case, however. I'm pretty sure what the cause of these effects is. Thanks for your description, I'll try to fix it in the next version.

I also got the same exact problem. I'd just like to add that I use the Loop a lot... Other than that, very nice peace of work

AAC @ 128kbps listening test discussion

Reply #247 –
About considering ranked references as 5.0, as the ranked score, or drop them...

Here is the rationale Garf sent me:

Quote
Ranking a reference means that you underestimated the codecs performance on that sample.

Consider the following: you ABX it 110/200. That's a significant result, meaning you hear a difference. But you can't really tell the encoded one from the original, can you? Certainly reasonable to give 5.0 then.

Nice paradox


Here is the one Schnofler sent me:

Quote
Intuitively, I would either count the rating of the reference as the encoded sample's rating (the argument being that choosing sliders is just one more ABX trial) or discard the rating altogether (this being the conservative tactic, because the first method might be regarded sleazy). But as the listener obviously didn't intend to rate the sample as transparent and furthermore proved that it is in fact not transparent to him, your solution isn't immediately clear to me.


As you can see, two very good rationales, and both conflict.

What I will probably do is: at first consider only the "clean" results. If there are enough of these, the official test results will come from them. If not, I'll throw in the ranked references. I guess that's the best way to make everyone happy - otherwise, there would be no test results.

Now, the big question is, if I use the ranked references, should I use the ranked score, or grant a 5.0 score to them?

Please discuss.

Regards;

Roberto.

AAC @ 128kbps listening test discussion

Reply #248 –
My method:
If there's an ABX result with (number of trials)+1 on the margin (<0.1 but >0.05), treat the result as 5.0.
Else throw out the ranking.

Roberto, when will the test end exactly? Do you have enough results?
ruxvilti'a

AAC @ 128kbps listening test discussion

Reply #249 –
Quote
Now, the big question is, if I use the ranked references, should I use the ranked score, or grant a 5.0 score to them?

Assuming the ranked references are spread equally over the samples, this will of course affect the ratings, but also equally.
So maybe it's unimportant, how to rate.
What are the exact values of the ratings good for? The ratings of different listening-tests can't be directly compared (with a very bad anchor you tend to rate the other samples higher).