I've wrote a program for evaluating the corrected p-val for situation 2. It takes the maximum allowed number of ABX trials and the required confidence (i.e. traditional p-val) as arguments and returns the probability to pass the test with guessing.
Example:
(4, 0.95) -> 0 (impossible)
(5, 0.95) -> 0.03125 (5/5 -> pval=.96875)
(6, 0.95) -> 0.03125 (either the test is won at 5/5 or lost)
(16, 0.95) -> 0.08755493164
(100, 0.95) -> 0.2020580977 (!!!)
(16, 0.99) -> 0.01422119141
The idea is:
Construct a pascal triangle up to a certain level.
1
1 1
1 2 1
1 3 3 1
1 4 6 4 1
1 5 10 10 5 1
....
Now we can read it as follows: A(row=trials+1, column=correct+1) / 2^trials = P(abx=correct/trials | trials used) (e.g. the probability to score 2 times correct out of 3 is A(4, 3)/2^3 = 3/8)
The pval would be the sum of all those probabilities to the left of the chosen item.
My program does the following: seek for the earliest win condition (e.g. 5/5), calculate its probability (e.g. 0.03125) and set the corresponding item in the pascal triangle to 0 and recalculate, e.g.:
1 5 10 10 5 0
1 6 15 20 15 5 0
....
The last step is to make sure, that nothing is counted twice. Now start again. The next win condition with probability /= 0 is 7/8, with remaining probability of P(abx=7/8 | not 5/5) = 0,0195.., and so on...
Here's the Maple source code, I hope it's readable (# starts a comment):
CorrPVal:=proc(n,reqConfidence,Prob)
local k, Trial, LastResult, Result, Confidence:
Result:=array([1,seq(0,i=1..n)]): # initialize [1,0,...,0]
Prob:=0:
for Trial from 1 to n do # create new line of triangle
LastResult:=copy(Result): # only the last lines is required, so nothing
# more is stored / copy to help variable
Confidence:=0:
Result[1]:=1: # set first element to 1
k:=1: # now the rest
Confidence:=Confidence+binomial(Trial,0)*1/2^Trial:
while Confidence < eval(reqConfidence) and k <= Trial do
# check if target confidence is reached or all trials has been attempted
Result[k+1]:=LastResult[k]+LastResult[k+1]:
# calculate new element of Pascal triangle
Confidence:=Confidence+binomial(Trial,k)*1/2^Trial:
# increase Confidence (->more trials were correct)
k:=k+1:
end do:
if k<=Trial then # winning condition
Prob:=evalf(eval(Prob)+(LastResult[k]+LastResult[k+1])/2^Trial):
# add to sum of all winning probabilities
end if:
end do:
end proc:
# now follows the execution
# the result is stored in variable 'prob'
CorrPVal(16,0.95,prob):
prob;
# the result is displayed