Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Random Numbers for ABX Testing (Read 3970 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Random Numbers for ABX Testing

From Pio2001's very interesting post on designing a test to determine the audible effect of 24/96:
Quote from: Pio2001 on
The most secure way to get random numbers, in my opinion, is to use dices, with dice cups (like these : http://www.bgshop.com/ )

Is this really necessary? Sure, I agree with you completely that good quality random numbers are necessary to ensure a good test. Basically, my question is this: is using real random numbers (as opposed to PRNG generated "randomness") necessary to ensure that a test is valid?

I would expect that the output of a good PRNG, seeded with a decent seed, would be random enough that knowledge of the pattern of the random numbers would not bias a test. For example, the Mersenne Twister algorithm with the parameters set to decent values is well distributed in the first 623 dimensions. I haven't seen any evidence that a human would do better than chance at predicting the next value in the line. While lower quality PRNGs could be a problem - good quality PRNGs are widely available with several free implementations available. One good implementation is from Boost.

If there is a concern about PRNG derived values, then high quality randomness can still be obtained more easily than by throwing dice. www.random.org would be one source (and there are more on the web). A hardware RNG would also be easier to use in the test environment than dice throwing.

Random Numbers for ABX Testing

Reply #1 –
???
It must have been a joke. I mean, it's true that for good random you need a source from outside the computer... But dice? Talk about low tech, not to mention arduous. Anyways, if your random is good enough for crypto, it's good enough for an abx test.

Actually, I think his post was in the context of manually testing outside of a computer, with hifi equipment. So the person running the test would need have a random source... Though it still might be easier to pre-generate a random sequence ahead of time on the computer and print it out. Anything you can do to minimize the time between samples of the test is a good thing.

...
Random.org is pretty cool. There's also lavarnd, but they don't use a real lava lamp anymore so they're not nearly as cool as the old sgi lavarand was. That was pretty funny when I first saw it, back in the days when webcams were the in thing.

Random Numbers for ABX Testing

Reply #2 –
that was most certainly a joke (or he was drunk). you don't really need random numbers there, the only thing you'll have to do is reorder sample so that no one can figure out a pattern, a cheap pseudo random generator (even the craptacular rand in unix, which is totally predictable if you observe enough rolls) will do fine.

the only time you need real randomization in statistics is when you have a huge pool of samples, and you pick randomly out of that list, like 100 or whatever samples you'd like to test, this way you ensure that different characteristics of sound are captured evently (from classical to electic) even characteristics you are not aware of will be equally picked by this method. Nothing else needs real randomization.

once you picked the samples for your test, their ordering (though probably should change to avoid placebo) doesn't have to be random.
The Plan Within Plans

Random Numbers for ABX Testing

Reply #3 –
In hifi hardware tests, we don't have computers at hand.

Choosing ourselves a given sequence is something to avoid. The human thinking have some preconceptions about randomness that makes some choices more probable than other.

The use of dices was suggested in order to avoid coin flipping, that can be completely biased according to the way you flip it. For example, if you stand the coin slightly bend backwards and make it turn with a flick, most of the times, it will fall on the side that was in front of you. If then you just pull it from its prevous position, you get a mostly constant sequence of results from coin flipping.

Random Numbers for ABX Testing

Reply #4 –
Quote from: lexor on
that was most certainly a joke (or he was drunk). you don't really need random numbers there, the only thing you'll have to do is reorder sample so that no one can figure out a pattern, a cheap pseudo random generator (even the craptacular rand in unix, which is totally predictable if you observe enough rolls) will do fine.
This one depends a lot on how you use it. For example, some implementations of rand() use a generator from Numerical Recipes in C that alternate between odd and even (that is, the LSB alternates between on and off), the second least significant bit also has some pattern to it, with a period of about 30. A bad implementation that uses these bits (like X=(rand() & 0x1) would easily bias a test. Individual bits from these sorts of generators also exhibit a fair amount of skew (either 1 or 0 is more likely than the other), making either A or B more likely. While skew isn't likely to produce a false positive, it does mean you have to calculate p differently to get the right answer.

On an aside - Von Neumann's transition mapping algorithm for removing skew is one of the cleverest algorithms I have ever seen.
Quote from: Pio2001 on
The use of dices was suggested in order to avoid coin flipping, that can be completely biased according to the way you flip it. For example, if you stand the coin slightly bend backwards and make it turn with a flick, most of the times, it will fall on the side that was in front of you. If then you just pull it from its prevous position, you get a mostly constant sequence of results from coin flipping.
Ok, thanks, I understand now. I originally thought you were making a profound statement about the pattern prediction ability of test subjects 

If you were really obsessive you could use casino dice and a new die for each roll.