IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
Calculating the P-Value
torok
post Oct 30 2003, 01:38
Post #1





Group: Members
Posts: 391
Joined: 24-December 02
From: Eugene, OR
Member No.: 4224



Does anyone want to give me the formula? Thanks. biggrin.gif


--------------------
http://www.pkulak.com
Go to the top of the page
+Quote Post
phong
post Oct 30 2003, 04:23
Post #2





Group: Members
Posts: 346
Joined: 7-July 03
From: 15 & Ryan
Member No.: 7619



I was looking for the same thing myself... And I found it. I don't think I can insert mathML here, and I think code is a much better explanation than stupid mathy symbols, sere's some code I wrote minus some comments. It could be written to be much faster, but it is plenty fast enough for the number of trials we'd be talking about. Can you read Python?
CODE
def factorial(x):
   if x < 0: raise ValueError("Factorials for numbers less than zero are undefined")
   if x <= 1: return 1
   return reduce(operator.mul, xrange(2, x+1))

def binomial(n, k, p=0.5):
   if k > n:
       raise ValueError("Number of occurances (%d) exceeds number of trials (%d)." % (k, n))
   q = 1.0 - p
   return pow(p, k) * pow(q, n-k) * factorial(n) / (factorial(k) * factorial(n-k))

def abxBinomial(trials, correct):
   if correct > trials:
       raise ValueError("Number of correct ABX trials (%d) exceeds total number of trials (%d)." % (correct, trials)
   return reduce(operator.add, map(lambda k: binomial(trials, k), range(correct, trials+1)))


Quick explanation:
factorial() does exactly what its name says. It computes the factorial of a number.
binomial() uses the binomial distribution function to compute the probability that there would be exactly k occurances of some event of probability p occuring in n trials.
abxBinomial() computes the ABX pval you want. It takes the sum of the binomial function for correct, correct+1 ... num_trials.

This method is fine for use on a computer/calculator, but if you're working it out by hand there are formulas which approximate the value more quickly (if less accurately). Also, be aware that it overflows doubles if you take 170+ trials.


--------------------
I am *expanding!* It is so much *squishy* to *smell* you! *Campers* are the best! I have *anticipation* and then what? Better parties in *the middle* for sure.
http://www.phong.org/
Go to the top of the page
+Quote Post
sshd
post Oct 30 2003, 05:32
Post #3





Group: Members
Posts: 210
Joined: 16-June 03
Member No.: 7218



QUOTE (phong @ Oct 30 2003, 04:23 AM)
whatever

I admint I am not an expert, but the so-called P-value probably refers to the Greek letter Rho, which looks pretty much like a P.

Rho stands for probability.

It is quite easy to calculate the value. Say you have a normal 6 edged dice. The probability (Rho) of getting a 6 in the first try is 1/6 ~ 16,7%. The probability (Rho) of getting 6 in the fist try and 2 in the second try is 1/6 * 1/6 ~ 2,7%.

When you are doing simple ABX tests with only two possible results use 1/2 instead of 1/6.
Go to the top of the page
+Quote Post
ff123
post Oct 30 2003, 06:21
Post #4


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



The simplest way to do it is to simulate it:

http://ff123.net/abx/abx.html

You don't have to remember, derive, or look up all those messy equations. Plus, that's the way things happen in real life anyway.

ff123
Go to the top of the page
+Quote Post
torok
post Oct 30 2003, 07:28
Post #5





Group: Members
Posts: 391
Joined: 24-December 02
From: Eugene, OR
Member No.: 4224



QUOTE (sshd @ Oct 29 2003, 08:32 PM)
QUOTE (phong @ Oct 30 2003, 04:23 AM)
whatever

I admint I am not an expert, but the so-called P-value probably refers to the Greek letter Rho, which looks pretty much like a P.

Rho stands for probability.

It is quite easy to calculate the value. Say you have a normal 6 edged dice. The probability (Rho) of getting a 6 in the first try is 1/6 ~ 16,7%. The probability (Rho) of getting 6 in the fist try and 2 in the second try is 1/6 * 1/6 ~ 2,7%.

When you are doing simple ABX tests with only two possible results use 1/2 instead of 1/6.

I'm pretty sure that's not it. The P-value is finding the odds that something occured by chance alone, without any other external factors.


--------------------
http://www.pkulak.com
Go to the top of the page
+Quote Post
Continuum
post Oct 30 2003, 08:22
Post #6





Group: Members
Posts: 473
Joined: 7-June 02
Member No.: 2244



QUOTE (torok @ Oct 30 2003, 08:28 AM)
I'm pretty sure that's not it. The P-value is finding the odds that something occured by chance alone, without any other external factors.

The p-value is the probability that the test is passed under the condition that you were only guessing. It is not the probability that you were guessing.
(Not sure if you meant this anyway.)


And if you don't speak Python (like me):

CODE
n! := 1*2*3*...*n  (n>0)
0! := 1

                      a!
binomial(a,b) := -----------  (a>=b)
                  b! (a-b)!

          trials
           ---     binomial(trials,k)
p-val :=   \      --------------------
           /              trials
           ---          2
        k=correct


This post has been edited by Continuum: Oct 30 2003, 08:23
Go to the top of the page
+Quote Post
Pio2001
post Oct 30 2003, 13:24
Post #7


Moderator


Group: Super Moderator
Posts: 3936
Joined: 29-September 01
Member No.: 73



QUOTE (Continuum @ Oct 30 2003, 08:22 AM)
The p-value is the probability that the test is passed under the condition that you were only guessing.

Why does the old PCABX program gives a 100% probability for one success out of one test ? It should be 50 %.
Go to the top of the page
+Quote Post
Continuum
post Oct 30 2003, 14:50
Post #8





Group: Members
Posts: 473
Joined: 7-June 02
Member No.: 2244



QUOTE (Pio2001 @ Oct 30 2003, 02:24 PM)
QUOTE (Continuum @ Oct 30 2003, 08:22 AM)
The p-value is the probability that the test is passed under the condition that you were only guessing.

Why does the old PCABX program gives a 100% probability for one success out of one test ? It should be 50 %.

No idea, I never used it. 50% is correct.

I assume the calculations of PCABX are correct for more trials?
Go to the top of the page
+Quote Post
StoneRoses
post Oct 30 2003, 15:02
Post #9





Group: Members
Posts: 97
Joined: 19-September 03
From: BKK, Thailand
Member No.: 8916



Continuum's definition of P-Value for ABX test is correct. (binomial test; h0: you are guessing [p=0.5], h1: you are not guessing )

And I think PCABX just use wrong formular.
Go to the top of the page
+Quote Post
schnofler
post Oct 30 2003, 18:35
Post #10


Java ABC/HR developer


Group: Developer
Posts: 175
Joined: 17-September 03
Member No.: 8879



Just for the sake of completeness, here's the method I use in abchr/java:
CODE
public static double pVal(int n,int correct) {
    double result=0;
    double binom;
    int twos;
    for(int k=correct;k<=n;k++) {
 twos=n;
 binom=1;
 for(int j=0;j<k;j++) {
     binom*=(n-j)/(double)(k-j);
     while(binom>2 && twos>0){binom/=2;--twos;}
 }
 for(int j=0;j<twos;j++){binom/=2;}
 result+=binom;
    }
    return result;
}

Conceptwise, it's exactly the same as the method phong uses, although not as straightforward. The advantage is mainly that it's probably a bit faster and it doesn't overflow during your casual 170+ trials ABX test biggrin.gif
I don't know exactly when it does overflow, but I tested it once for 10000 trials with 5000 correct and it got the right result after about 5 minutes of calculating (well, this is a scientific forum, you know.. every last 0.0001% of certainty in an ABX test is valued tongue.gif )
When it comes to optimizing programs I'm a total amateur, though (I rarely do much optimizing in my programs), so there are probably much better ways to calculate that.

This post has been edited by schnofler: Oct 30 2003, 18:36
Go to the top of the page
+Quote Post
ff123
post Oct 30 2003, 19:10
Post #11


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



In my version of abchr, I don't bother calculating something if it's not going to change the final answer by a certain amount (p values < 0.001 are not reported to their full precision). This skirts both the underflow and speed issues.

ff123
Go to the top of the page
+Quote Post
schnofler
post Oct 30 2003, 19:24
Post #12


Java ABC/HR developer


Group: Developer
Posts: 175
Joined: 17-September 03
Member No.: 8879



QUOTE
In my version of abchr, I don't bother calculating something if it's not going to change the final answer by a certain amount (p values < 0.001 are not reported to their full precision). This skirts both the underflow and speed issues.

Well, that might be true. But as a math student, I take pride in considering this unprofessional. wink.gif
Go to the top of the page
+Quote Post
ff123
post Oct 30 2003, 21:26
Post #13


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



QUOTE (schnofler @ Oct 30 2003, 10:24 AM)
QUOTE
In my version of abchr, I don't bother calculating something if it's not going to change the final answer by a certain amount (p values < 0.001 are not reported to their full precision). This skirts both the underflow and speed issues.

Well, that might be true. But as a math student, I take pride in considering this unprofessional. wink.gif

Haha! Since I'm an engineer, I take that as a compliment.

ff123
Go to the top of the page
+Quote Post
schnofler
post Oct 30 2003, 22:18
Post #14


Java ABC/HR developer


Group: Developer
Posts: 175
Joined: 17-September 03
Member No.: 8879



QUOTE (ff123 @ Oct 30 2003, 12:26 PM)
QUOTE (schnofler @ Oct 30 2003, 10:24 AM)
QUOTE
In my version of abchr, I don't bother calculating something if it's not going to change the final answer by a certain amount (p values < 0.001 are not reported to their full precision). This skirts both the underflow and speed issues.

Well, that might be true. But as a math student, I take pride in considering this unprofessional. wink.gif

Haha! Since I'm an engineer, I take that as a compliment.

ff123

LOL! laugh.gif
Who said there can be no holy wars in a perfectly scientific discussion? biggrin.gif
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 21st September 2014 - 16:25