IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
ABX questions, "hide results" option
LoFiYo
post Mar 27 2004, 18:21
Post #1





Group: Members
Posts: 133
Joined: 2-January 04
Member No.: 10896



Hi, I have a few quick questions to knowledgeable people here about the Hide Results button on computer ABX programs.

1) What was the original intention of the Hide Results button?

2) Should it be a personal choice whether to hide the results or not?

3) Which gives fairer results, to hide the results or not?

4) In properly designed listening tests, are the results hidden or shown during the test?

5) (off-topic) Shouldn't FILEABX be listed in the FAQ section as an ABX program?
Go to the top of the page
+Quote Post
Continuum
post Mar 27 2004, 18:39
Post #2





Group: Members
Posts: 473
Joined: 7-June 02
Member No.: 2244



QUOTE (LoFiYo @ Mar 27 2004, 06:21 PM)
3) Which gives fairer results, to hide the results or not?

Depends on how your test is set up: Do you fix the number of total trials before you start the test, or do you decide when to stop based on your results? In the first case both tests yield the same statistical significance (showing the results is of no use to a guessing listener), while in the second case the confidence value is a little skewed.

(There are, however, ways to compensate for this effect, see "profiles" in the more or less recent statistics threads.)
Go to the top of the page
+Quote Post
tigre
post Mar 27 2004, 18:59
Post #3


Moderator


Group: Members
Posts: 1434
Joined: 26-November 02
Member No.: 3890



The point about hiding results during the test is this:

The p-values all recent ABX programs show, only give you the correct "probability to reach the current score (or better) by guessing" if the number of trials has been fixed before the test starts. If the tester is allowed to stop the test whenever he likes (e.g. when a certain p-value is reached), the real "probability ..." is bigger. I tried to explain this in the 1st post of this thread.

One way to avoid this is to force the tester to enter the number of trials he's going to perform at the beginning of the test, another one is to hide the results.

So the advantage of hiding the results is that the p-values are correct, the disadvantage is (obviously) that the tester doesn't have feedback and e.g. might not notice that he becomes fatigued quickly enough.

A rule-of-thumb workarround would be to "perform one trial extra", i.e. if you want to reach a "probability ..." of 0.01, you don't stop after reaching 7/7 or 10/11 (-> p-value < 0.01) but perform one additional trial, i.e. you stop if 8/8 or 11/12 is reached. This method works good enough for < 30 trials.

A safer method would be to use the little program I uploaded here to calculate "stop points" before you start the test (or use the examples from the post), e.g.:
You want to reach a "probability ..." of 0.01. Now depending on how the stop points are calculated (you have the choice - are you aiming for quick success = 7/7 or do you expect some hard fight scoring e.g. 36/50 in the end?), the test will be finished successfully e.g. at
QUOTE
1. Stop point: (8/8) C-Value: 0.00390625
2. Stop point: (11/12) C-Value: 0.00585938
3. Stop point: (13/15) C-Value: 0.00769043
4. Stop point: (16/19) C-Value: 0.00844574
5. Stop point: (18/22) C-Value: 0.00913858
6. Stop point: (21/26) C-Value: 0.00942713
7. Stop point: (23/29) C-Value: 0.00969638
8. Stop point: (26/33) C-Value: 0.00981075
9. Stop point: (29/37) C-Value: 0.00986504
10. Stop point: (31/40) C-Value: 0.00991874
11. Stop point: (34/44) C-Value: 0.0099426
12. Stop point: (36/47) C-Value: 0.00996601

Or you give up without reaching one of those scores and the test is failed.

I hope this answers most of your questions, otherwise feel free to ask.


--------------------
Let's suppose that rain washes out a picnic. Who is feeling negative? The rain? Or YOU? What's causing the negative feeling? The rain or your reaction? - Anthony De Mello
Go to the top of the page
+Quote Post
LoFiYo
post Mar 28 2004, 04:04
Post #4





Group: Members
Posts: 133
Joined: 2-January 04
Member No.: 10896



Thanks guys for your replies. smile.gif
Go to the top of the page
+Quote Post
LoFiYo
post Mar 28 2004, 06:32
Post #5





Group: Members
Posts: 133
Joined: 2-January 04
Member No.: 10896



One more quick question while we're at it tongue.gif

There are two flavors of WinABX. There is ABX and there is ABA. I think ABA might be better when it comes to avoiding fatigue influence, because with ABA you can reach the target p-value quicker than ABX, therefore less fatigue factor. What do you think?
Go to the top of the page
+Quote Post
ff123
post Mar 28 2004, 06:48
Post #6


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



QUOTE (LoFiYo @ Mar 27 2004, 09:32 PM)
One more quick question while we're at it  tongue.gif

There are two flavors of WinABX. There is ABX and there is ABA. I think ABA might be better when it comes to avoiding fatigue influence, because with ABA you can reach the target p-value quicker than ABX, therefore less fatigue factor. What do you think?

Possibly, although you might have to put in more effort to figure out which sample is the odd man out. For obvious artifacting, it is more efficient than ABX, but then again, there isn't much use in ABX'ing obvious artifacts anyway. It's debatable whether or not this is more effective for defects which are at the edge of perception. Try them both and decide which way works best for you.

ff123
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 25th December 2014 - 12:40