2
$\begingroup$

I am using NIST Test suite on a PRNG and have a confusion regarding some specific results. Following is the part of the results:

RESULTS FOR THE UNIFORMITY OF P-VALUES AND THE PROPORTION OF PASSING SEQUENCES

 C1  C2  C3  C4  C5  C6  C7  C8  C9 C10  P-VALUE  PROPORTION  STATISTICAL TEST
  0   0   0   0   1   2   3   3  15  76  0.000000 *  100/100     BlockFrequency

My question is what is the significance of above result when p-value is 0.0 for Block Frequency Test but proportion is showing 100/100 test sequences passed the test and should I consider this test to be passed of failed? I have checked other questions regarding the NIST Test Suite but they don't answer this specific query.

Improve this question
$\endgroup$
1
$\begingroup$

Randomness is pesky, and can wax and wane between samples. That's why there is no one specific categorical test that can validate an IID sequence. You need to statistically combine several (perhaps contradictory ones) together to gain a broader perspective. The P values in columns C1 thru C10 are distributed as:-

distribution

Each column is a P value of width 0.1. So what seems to have happened is that all your individual P values fell 0.4 < P < 0.99, which is the NIST 0.01 significance level. So individually they passed, hence 100/100.

But when you plot this distribution of P values, it looks unlikely. The test suite is designed so that all of the P scores are uniformly distributed 0 - 1 under an is-random null hypothesis. And when you calculate a probability of these scores occurring randomly (via a Chi test), you get your P = 0.000000 (plus star!). So you failed the test. Badly :-(

There's more interpretative stuff in §5 of the Special Publication 800-22 document. I won't say any more regarding your PRNG as there is limited information in this question.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I am now clear with what these two values viz. the p-value and the proportion represents. But, if I have to claim whether this PRNG failed/passed on this particular test, what should I claim? I went through the NIST document, they didn't specify such cases there. $\endgroup$
    – Lambda
    Jul 12 '19 at 15:23
  • 1
    $\begingroup$ @Ashutosh It failed prima facie. This comes down to my opening sentence that a little experience is required. A 1% significance level would suggest 1% of the tests failing in a random situation. That simply means P < 0.01. You got 0.000000, and experience suggests that when a generator has an inherent problem it will fail badly. Your's failed ever so badly. So, I would rerun the test with another independent data set and see. Data's cheap. There's a chance it will pass repeated tests, but currently your's has failed this randomness test. That's all I can say as randomness is, tricky. $\endgroup$
    – Paul Uszak
    Jul 12 '19 at 16:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.