I have a sequence of integers between 0 to 100. Is there any free software like Sage, where I can test randomness of this sequence? I have Linux machine.
I have huge data set. Actually I have my own generator. I want to test its randomness.
Cryptography Stack Exchange is a question and answer site for software developers, mathematicians and others interested in cryptography. It only takes a minute to sign up.
Sign up to join this communityI have a sequence of integers between 0 to 100. Is there any free software like Sage, where I can test randomness of this sequence? I have Linux machine.
I have huge data set. Actually I have my own generator. I want to test its randomness.
Tests of randomness with only data as input can give proof of non-randomness, but never a credible indication of randomness unless their result is coupled with an analysis of how the random data tested has been generated. Without such knowledge, such tests give a falsely reassuring PASS, or a FAIL.
Illustration: consider the PRNG that outputs 512-bit blocks computed as the HMAC-SHA-512 of the previous block under some key. That pass any randomness test for one not knowing the key, yet is trivially predictable from past output with that knowledge.
In cryptography, randomness tests with PASS result can only be useful when and if we have a model of the source tested. This is at the heart of the AIS31 methodology of Common Criteria evaluation for True Random Numbers Generators in things like Smart Cards; see there (under AIS31; the page exists in German only AFAIK, but has links to many documents in English and a Reference implementation of the statistical tests).
Per the AIS 31 methodology, it is made some model matching the device, and justified that per that model, any likely defect that do not raise alarm won't result in using a significantly predictable bitstream. Typically there is:
I take it that this has a connection with cryptography, as some of the other SE forums might be able to help with programmatic implementation.
However. With a number range of 0-100, all you'll be able to do is a Chi Squared test to obtain a probability (0-1) that it could occur randomly. This is to check the frequency of occurrences of the 0 -100 values throughout your data set. They should all be even(ish). See Wikipedia for it's use. It's fairly simple. If you rerun the test a 100 times, getting all sorts of p values from 0.0000 - 1.0000, you can then use a Kolmogorov Smirnov (KS) test on those p values to get another p value(!). This does somewhat average out the anomalies from the first where you will get some failures like p=0.0001. I find that commons.math has an easy Java implementation of KS. It also does Chi tests, but I suggest in that case you perform G tests as they're preferred these days.
As you have access to the generator, you might be able to convert it's to output a 0-255 range. That then means you can use some standard tests. I recommend:-
RNGTEST for files in 20Kbit blocks
DIEHARD for files = 10MB
ENT for files 500KB - 2GB
DIEHARDER for files > 10GB(ish)
All these packages can take a feed from standard output so it's easy to connect them to your generator if you have source code access. They are easily downloadable as complied packages for Linux.I wouldn't bother with any other tests such as TESTU01 as they're unreliable in the extreme. And I also suggest that you don't try to write you own versions of theoretical tests unless your can also test that your test code works properly. And what would you test it with?
(How do you manage to get a native 0-100 output from a numeric generator???)