I do not understand this at all. If you have a sighted test where you believe the two items under test ought to sound identical (but in fact they don't) - surely that could quite easily generate false negatives?

Nope. I've multiple times made a perfect score by just randomly clicking. Now imagine what a spectrum analyzer does...

An online ABX test only works if you have honest participants that will not only not cheat but also point out and accept problems with the test files (like the time offset in the AVS AIX test files).

Nope. You really should read up on statistics again.

What is a "real" difference? A measurable difference certainly does not mean that there's an audible difference anyway.

And nope, there is concern given to false negatives, for example by including low anchors in test files.

But again, in an online test you can only assume that people try their best and also list the equipment they actually used.

If this is not the case then it still does matter less than false positives, because we do not accept the null hypothesis anyway. Again, read up on statistics.

It certainly takes faith to accept (positive) results of demonstrably dishonest people.

But that aside, it seems you are trivializing this. Black/white kinda thinking, as you did above.
How would an online test look like where you can calculate specificity for each participant? I would really be interested in your answer.

It's hard enough (= impossible) to get honest people doing the required number of trials and sending in their results regardless of success.


Nope.
No faith required.

Where you need faith, or let's better call it gullibility, is with the (most of the time) positive dishonest sighted listening tests.

Quote from: jkeny on 2015-02-01 21:25:29

This is as ridiculous a statement as I have heard & shows your lack of knowledge & understanding - there's no such thing as a false negative in sighted tests

I do not understand this at all. If you have a sighted test where you believe the two items under test ought to sound identical (but in fact they don't) - surely that could quite easily generate false negatives?

Just a trivial answer to to point out the stupidity of such a claim: simply not properly entering the intended choice can result in a false negative (or a false positive).

Regardless, CC's post was more thought provoking.

A false negative is a true positive called as a negative. What's your definition?

And how do you reconcile *demanding* adherence to e.g., ITU standards from everyone else, with your championing of your and Amir's decidedly non-compliant methods?

ABX removes bias that could produce false positives but it does not remove bias towards false negatives. Subjects biased towards not hearing a difference can throw off your results by not listening and guessing randomly.

You've many times got a perfect 20 out of 20 correct  trials by guessing? I believe you should immediately buy a lotto ticket as you are on eof the luckiest people I have ever met. Can you tell us th eprobablility of doing what you claim you did?

Yep, I guess we have seen an example of Arny cheating with his ABX results - just random guesses but it takes a lot more effort to cheat & get an overall positive score. I'm sure there might be some bothered to go to this trouble but I doubt there are many.

Yea, yea, it's a confidence level - so what?

A "real" difference is an agreed audible difference that is recognised - such as are already established in JNDs

what low anchors are included inside an ABX test?

Could you express that better - it's cryptic

Quote

How would an online test look like where you can calculate specificity for each participant? I would really be interested in your answer.

Easy enough -  you would have known audible differences randomly added to one of the files during the listening trials & the ABX software would record this trial result as a control result. So if this difference was not identified then a false negative is recorded. The analysis of such stats for many tests would be interesting

That's the usual answer given - "Huh, our tests might be bad & unreliable but look at the sighted test, it's worse". Pretty scientific attitude
Again, if you guys aren't interested in the sensitivity of your tests then why should anybody pay any credence to the results?

Quote from: Arnold B. Krueger on 2015-01-31 12:23:09

Quote from: ajinfla on 2015-01-31 11:43:39

I know you dismiss any ABX test lacking positive controls/ hidden reference anchors,

I agree that given their propensity for false negatives (IME a small price to pay for the much-needed control over false positives), listening tests need positive controls.

This is typical of the skewed approach to audio DBTs - you are happy that they are a small price to pay but you have no idea how skewed the results are towards false negatives.

That's the usual answer given - "Huh, our tests might be bad & unreliable but look at the sighted test, it's worse". Pretty scientific attitude
Again, if you guys aren't interested in the sensitivity of your tests then why should anybody pay any credence to the results?

Sighted tests are not worse, they are completely worthless for determining small audible differences. They are not a worse alternative, they are not even an alternative.

I do not think that e.g. ABX is bad or unreliable at all - if you have honest participants.

In the beginning we underestimated the difficulty of doing good listening tests simply because we gave too much (any at all) credibility to sighted evaluations which were the standard up until that point.

Quote from: Arnold B. Krueger on 2015-01-31 12:23:09

Quote from: ajinfla on 2015-01-31 11:43:39

I know you dismiss any ABX test lacking positive controls/ hidden reference anchors,

I agree that given their propensity for false negatives (IME a small price to pay for the much-needed control over false positives), listening tests need positive controls.

This is typical of the skewed approach to audio DBTs - you are happy that they are a small price to pay but you have no idea how skewed the results are towards false negatives. So what if 90% of audio DBTs were found to be suffering from an unacceptable level of false negatives - would this be a small price?

Quote

I have problems with people who rant and rave about this issue as an ABX-only problem when it is inherent in any listening test.  The only reason why nobody says much about the false negatives in sighted evaluation is that they are washed out by the very many false positives.

This is as ridiculous a statement as I have heard & shows your lack of knowledge & understanding - there's no such thing as a false negative in sighted tests 

Quote

I have more problems with people who won't recognize that false negtatives are a problem that is easy enough to manage.

yes, it's easy to manage by including hidden controls within the test but have you ever done so & can give us the stats on false negatives for any such test you've administered? I don't even think I have ever seen you give a practical, sensible approach to how these controls could be included in a test?

Quote from: jkeny on 2015-02-02 03:31:32

You've many times got a perfect 20 out of 20 correct  trials by guessing? I believe you should immediately buy a lotto ticket as you are one of the luckiest people I have ever met. Can you tell us the probability of doing what you claim you did?

Nope, but nice trolling.
I've done enough trials to push the probability of guessing below 5%. If the probability is just below 5% then you only need 20 tries to get - according to you - a true positive result, which is of course nonsense. For a perfect score you obviously need more luck, and much more luck if you want to do 20/20. Or you simply cheat. 100% probability for any number of trials.

Quote from: jkeny on 2015-02-02 03:31:32

That's the usual answer given - "Huh, our tests might be bad & unreliable but look at the sighted test, it's worse". Pretty scientific attitude
Again, if you guys aren't interested in the sensitivity of your tests then why should anybody pay any credence to the results?

Not at all.
Sighted tests are not worse, they are completely worthless for determining small audible differences. They are not a worse alternative, they are not even an alternative.

I do not think that e.g. ABX is bad or unreliable at all - if you have honest participants.

Your above test doesn't help much at all, because a cheater would always correctly identify. Well almost always, because introducing some error or external influence ("dog barked here", "wife interrupted here" ...) makes it look more real.

Quote from: jkeny on 2015-02-02 03:31:32

That's the usual answer given - "Huh, our tests might be bad & unreliable but look at the sighted test, it's worse". Pretty scientific attitude
Again, if you guys aren't interested in the sensitivity of your tests then why should anybody pay any credence to the results?

That's a false statement of the real situation.  The real situation is that science knows how to make DBTs the gold standard for reliability and have done so many times.

There is very common far worse answer: "Our tests are known to be biased and unreliable but we base just about everything we do on them and advise others to do the same".

That is the de facto position of most people who writer for or publish high end ragazines, sell high end audio gear, and in particular those who sell high end DACs.

ok so you want some tests inside the abx to control that the examinees isn't failing on purpose or out of boredom, bad faith... on principle why not. it wouldn't really be detrimental as long as it doesn't complicate stuff too much for the layman person to still use the test.

but how do you suggest to do that? personally I don't know how to do it. let's say the ABX makes 3files instead of 2, one with a slight EQ or a delay or added noise or whatever it is that people should be able to hear. then the guy of bad faith (Arny apparently is your best friend), would hear it, recognize it for what it is and answer correctly before going back to randomly clicking on the rest of the tracks just to make fun of you. I don't see how to create an automated system to foolproof the test against false negative? conscious or subconscious.

I have no problem thinking that abx isn't perfect, and in fact, most of the weaknesses of abx, I've learned from reading Arny himself warning against them. that's how much bad faith the guy puts on the internet to blindly promote ABX and make billions out of the abx plugin on foobar. Arny, the only guy on the net with a clear money agenda(sorry my sarcasm leaks out when I read too much BS).
anyway, at the moment I believe ABX is still one of the most effective and easily accessible way to do many subjective hearing tests. if you have a more reliable, free, and worldwide accessible method, please pretty please, tell me all about it. I'm all for the latest good stuff. as long as it's not some sighted listening middle age nonsense, I'm all for it.

All I'm asking for is the rate of false negatives in the test results. It shouldn't be seen as a threat to anybody if the test is reliable. So why is it seen this way?

So now you are going back on what you said "I've multiple times made a perfect score by just randomly clicking" - now your claim has changed - please make up your mind.

You are so hung up on the cheating to get positive results - why does ArnyK's cheating to get a null result not also apply - he didn't listen, just randomly hit keys?

Good - a rational voice, at last!

Quote from: jkeny on 2015-02-02 15:44:26

All I'm asking for is the rate of false negatives in the test results. It shouldn't be seen as a threat to anybody if the test is reliable. So why is it seen this way?

I don't think anyone is seeing this as a threat. However, you do not seem like taking into account what it would mean if the tests accounted for false positives and false negatives in the same way. If you had three of those tests, and (hypothetically) one would yield an overall confirmation, while two yield a rejection, then you would have to go for the majority.

If, on the other hand, the three tests didn't account for false negatives, and only attempted to control false positives, as was discussed here, then one successful test out of three would suggest that the null hypothesis can be rejected, albeit with perhaps a rather low confidence.

So my suspicion is that you want to have your cake and eat it. That's often the case when people with vested interests play the statistics interpretation game. There simply are a lot of possibilities to fool oneself without it being blatantly obvious.

You have no clue of the type II errors in your results - no clue how to run a test & yet you claim it's reliability & a gold standard. Give me a break!

Quote from: jkeny on 2015-02-02 15:29:08

So now you are going back on what you said "I've multiple times made a perfect score by just randomly clicking" - now your claim has changed - please make up your mind.

You should go back and read again what I said and what I did not.

Also, I wrote this not to demonstrate a problem with ABX protocol, but your lack of understanding of statistics.

Quote from: jkeny on 2015-02-02 15:33:05

You are so hung up on the cheating to get positive results - why does ArnyK's cheating to get a null result not also apply - he didn't listen, just randomly hit keys?

Please go back and read my previous replies including the suggestion to read up on hypothesis testing.

And the whole point of ABX is to reduce false positives, which in a perfect world would include all cheating attempts.