# Interpretation of the results of NIST (p)NRG suite

I have problems interpreting the NIST (sts-2.1.2) suite results. After running the statistics with 100 samples (each sample of 1000000 byte length) a new prng i got this result:

 ------------------------------------------------------------------------------
RESULTS FOR THE UNIFORMITY OF P-VALUES AND THE PROPORTION OF PASSING SEQUENCES
------------------------------------------------------------------------------
generator is <data/data.bin>
------------------------------------------------------------------------------
C1  C2  C3  C4  C5  C6  C7  C8  C9 C10  P-VALUE  PROPORTION  STATISTICAL TEST
------------------------------------------------------------------------------
13   9   8  11   8  11   5  14  12   9  0.678686     98/100     Frequency
12  11  11   7   9  11   8   8  10  13  0.946308     99/100     BlockFrequency
15   1  10  11   9  13  11  11   9  10  0.213309     99/100     CumulativeSums
13  10   7  10  11   9  15   6   8  11  0.678686     99/100     CumulativeSums
8  13   6  15  10   8   7  11  10  12  0.616305    100/100     Runs
11  10  11  14  13  10  10   6   8   7  0.779188     99/100     LongestRun
6   7  11  11   8   9   7  16  12  13  0.437274    100/100     Rank
13  11  13   8  12   6   7  11  10   9  0.798139     96/100     FFT
10  18   8   8   9   5  12  13  11   6  0.171867    100/100     NonOverlappingTemplate
9   9  14  12  12   7   9   9  10   9  0.924076    100/100     NonOverlappingTemplate
11   9   5   7  11   8   6  12  16  15  0.202268     98/100     NonOverlappingTemplate
5   7  12  10   9  14   9  11  15   8  0.474986     99/100     NonOverlappingTemplate
8   5  11   7   9  13  14   9   9  15  0.419021     99/100     NonOverlappingTemplate
12   8   8  11   8   9  10  14  12   8  0.897763     98/100     NonOverlappingTemplate
12  14  17  10   5   9   6   6  13   8  0.122325     98/100     NonOverlappingTemplate
8  10  14  14  11  15   8  11   4   5  0.171867    100/100     NonOverlappingTemplate
10   9  15  12   8   9  11   8   5  13  0.595549    100/100     NonOverlappingTemplate
7   7  15   8   6  11  13   7  13  13  0.350485    100/100     NonOverlappingTemplate
10  15   8   9  15   5  11   7  10  10  0.437274    100/100     NonOverlappingTemplate
12  12   9   9   6  11  13  11   7  10  0.867692    100/100     NonOverlappingTemplate
11  12  12   9  12   9  10   6  10   9  0.955835    100/100     NonOverlappingTemplate
9  16   8  15   2  12   8  11  14   5  0.035174     97/100     NonOverlappingTemplate
11   9  10   6  13   8  10   6  17  10  0.383827     99/100     NonOverlappingTemplate
9  13   8   9  10   9  15  10   7  10  0.834308     98/100     NonOverlappingTemplate
10  13  12   9   9  12  11   6  10   8  0.911413     99/100     NonOverlappingTemplate
11  11  10   9  11   7   6  15   9  11  0.779188     99/100     NonOverlappingTemplate
15   9  12   5   6  13  12   7  15   6  0.145326     99/100     NonOverlappingTemplate
12   9   8   9  13   6   7  15  13   8  0.514124    100/100     NonOverlappingTemplate
10  13  11  10   8  11  11   7   8  11  0.964295    100/100     NonOverlappingTemplate
11   9   7   9  12  13   8  12   8  11  0.924076     98/100     NonOverlappingTemplate
10  10   8  12   7  11  12  12   7  11  0.935716    100/100     NonOverlappingTemplate
8   9   9   8  18   9  11  11  10   7  0.474986    100/100     NonOverlappingTemplate
6  12   7   4   9  14   7   7  18  16  0.017912     99/100     NonOverlappingTemplate
14   9   7   7  14  11   9   9   8  12  0.719747     99/100     NonOverlappingTemplate
7   6  12  14   6  16   8   9  14   8  0.202268     99/100     NonOverlappingTemplate
10   9  14  11   3  11   8  13  10  11  0.514124     99/100     NonOverlappingTemplate
12   4  10  11  13   9   9  13   9  10  0.719747     98/100     NonOverlappingTemplate
12  13   5   7  14   8  14  11   8   8  0.419021     98/100     NonOverlappingTemplate
12   8   6   9  13   9  12  12   9  10  0.883171     99/100     NonOverlappingTemplate
11  11   7  14  16   6   8  10   9   8  0.455937     98/100     NonOverlappingTemplate
11  12  15  12  12   7  10   5   9   7  0.514124     98/100     NonOverlappingTemplate
8  11   5   7   9  12  14   6  14  14  0.289667    100/100     NonOverlappingTemplate
8   8  13   8  13   7   8  11  10  14  0.739918     98/100     NonOverlappingTemplate
14  12   8   9   7  10   8  11  13   8  0.816537     99/100     NonOverlappingTemplate
15   6   4  10   6   9   6  15  13  16  0.035174     97/100     NonOverlappingTemplate
4  10  13  11  13   6  12   9  10  12  0.534146    100/100     NonOverlappingTemplate
8  19   5  11  14  12   4   6   9  12  0.026948     98/100     NonOverlappingTemplate
9   7  15   8   8   9  21   7   9   7  0.030806     99/100     NonOverlappingTemplate
5  12  12  12  10   7   9   5  13  15  0.304126    100/100     NonOverlappingTemplate
10  13  16   6   9   6   7  11  10  12  0.419021    100/100     NonOverlappingTemplate
15   7  10   8  13  14   8   9  10   6  0.494392     99/100     NonOverlappingTemplate
5  13   6   9  13   8   8  13  12  13  0.437274    100/100     NonOverlappingTemplate
7  18  15   7   7  10   7   8   9  12  0.145326    100/100     NonOverlappingTemplate
8  12   8   7  13  13  15   7  13   4  0.224821     99/100     NonOverlappingTemplate
7  13  13  10   9  10  10   5  14   9  0.637119    100/100     NonOverlappingTemplate
11   5   7   7  14  12   9  12  14   9  0.474986     99/100     NonOverlappingTemplate
11  13   8   9  10   9  13   7  14   6  0.678686    100/100     NonOverlappingTemplate
9  13  10   8  10   9   9  13  10   9  0.978072     99/100     NonOverlappingTemplate
11  10  12   7   9  12  10   8  12   9  0.971699    100/100     NonOverlappingTemplate
10  11  11   3  16  12   8  11   9   9  0.366918     99/100     NonOverlappingTemplate
10   7  10  17   7   2   9  16  12  10  0.045675     99/100     NonOverlappingTemplate
6   7  14  14   8  10  15   9   5  12  0.236810     99/100     NonOverlappingTemplate
8   5   8  11   7   9  21  11  12   8  0.042808     98/100     NonOverlappingTemplate
9  11   9  14   9  12   5  12   7  12  0.678686     99/100     NonOverlappingTemplate
5   4  12  13   7   9  10  12  16  12  0.171867     99/100     NonOverlappingTemplate
12  10   9  10  16   9   7  10   7  10  0.739918     99/100     NonOverlappingTemplate
12  10  13   8  10  11   8  11  10   7  0.955835     99/100     NonOverlappingTemplate
10   8  12   7  15  15   4   8  10  11  0.289667     99/100     NonOverlappingTemplate
10   7  17   9  11  10   5   7  12  12  0.334538     99/100     NonOverlappingTemplate
11  12  19   9   8  13   8   7   8   5  0.115387     99/100     NonOverlappingTemplate
5  12   9   7  13  15  16   8   6   9  0.162606     99/100     NonOverlappingTemplate
13   9  13  10  10   7  12  11   9   6  0.834308    100/100     NonOverlappingTemplate
9  14  11   8  11   9   6   4  11  17  0.181557     99/100     NonOverlappingTemplate
4  10   6  13  10   7   8  13  12  17  0.137282     98/100     NonOverlappingTemplate
4  10  12  15  11  11   6  14   8   9  0.319084    100/100     NonOverlappingTemplate
11   8   6   8  15  13  11  11   6  11  0.554420     99/100     NonOverlappingTemplate
8   8   7   6  11   9  21  12   6  12  0.035174    100/100     NonOverlappingTemplate
14  11  11  12   6  17   5   8  10   6  0.153763    100/100     NonOverlappingTemplate
13   8  13   7  11   9  13  11   7   8  0.779188    100/100     NonOverlappingTemplate
6  13   7  10  14   8   8   7  13  14  0.419021     98/100     NonOverlappingTemplate
13   5   6   9   4  14  16   9  10  14  0.075719     99/100     NonOverlappingTemplate
15   8   8   9   9  10  11  11  10   9  0.924076     97/100     NonOverlappingTemplate
10  18   8   8   9   5  12  13  11   6  0.171867    100/100     NonOverlappingTemplate
16   9   9  10  12   5  15  11   6   7  0.224821     98/100     NonOverlappingTemplate
10   8  13  10   7   7   8  16  11  10  0.616305    100/100     NonOverlappingTemplate
11   6   9  10  10   9   5  15  13  12  0.514124    100/100     NonOverlappingTemplate
6   5   9  12  15  14  13   8   9   9  0.334538     98/100     NonOverlappingTemplate
5  16   5  11  11   7  12  12  10  11  0.304126    100/100     NonOverlappingTemplate
9   7  12   5  12  13   8  15   7  12  0.401199     99/100     NonOverlappingTemplate
10  12  10   7  11   9   8   6  16  11  0.616305     98/100     NonOverlappingTemplate
7  12   5   9  12  14   8  13  11   9  0.595549     99/100     NonOverlappingTemplate
10   8  13  10  11   9   8  19   8   4  0.122325    100/100     NonOverlappingTemplate
13  10  12   8   8  11   8  10   5  15  0.574903     99/100     NonOverlappingTemplate
10   8  10  10  12   8  10  12  12   8  0.983453    100/100     NonOverlappingTemplate
8  11  12  11  11  10   7  11   7  12  0.946308     99/100     NonOverlappingTemplate
9  14  11  10   7  11   9  10  10   9  0.964295     99/100     NonOverlappingTemplate
11  15   6  13  12   9   6   8   8  12  0.494392     97/100     NonOverlappingTemplate
5   7   8  12  16   8  13  11   9  11  0.401199     98/100     NonOverlappingTemplate
9  13  10   8  12   9  12   7   9  11  0.946308     98/100     NonOverlappingTemplate
7  11  15   8  10  17   9   7   7   9  0.289667     99/100     NonOverlappingTemplate
12   8  17   9  10  12  13  10   2   7  0.108791     99/100     NonOverlappingTemplate
7   9  12   7  10   9  15   9  14   8  0.637119    100/100     NonOverlappingTemplate
14  10  10   3   8  13  11  10   7  14  0.319084     99/100     NonOverlappingTemplate
6  11   9  11  10  17  10   7  14   5  0.224821    100/100     NonOverlappingTemplate
10  16   3   7  13   9   9   4  16  13  0.028817    100/100     NonOverlappingTemplate
15   7  10  16   4   9   7  15   9   8  0.102526     97/100     NonOverlappingTemplate
15   7  13   8   7  10   4  13  10  13  0.275709    100/100     NonOverlappingTemplate
8  12   9  16  10   6  12  13   7   7  0.419021     99/100     NonOverlappingTemplate
9  13  13  10  10   8   9   8   9  11  0.964295     99/100     NonOverlappingTemplate
9   6   8  14   8   9   8   7  12  19  0.122325    100/100     NonOverlappingTemplate
8  10  14  12  11  14  11  11   2   7  0.236810    100/100     NonOverlappingTemplate
9   9   9  11  10  13  12   9  10   8  0.987896     99/100     NonOverlappingTemplate
8  12  13  11   8   9   9   9   7  14  0.834308     96/100     NonOverlappingTemplate
10  10   5   8   8  10  13   8  12  16  0.474986    100/100     NonOverlappingTemplate
13  11   7  13   7  12  12   8   9   8  0.798139    100/100     NonOverlappingTemplate
9  13  17   5  10   5   8   6  15  12  0.071177    100/100     NonOverlappingTemplate
4  10  13  14  13  13  11   8   5   9  0.275709    100/100     NonOverlappingTemplate
10   9  10  11  10  10  12   9  14   5  0.851383     98/100     NonOverlappingTemplate
8  19   8  10  13   5  12   8   8   9  0.137282    100/100     NonOverlappingTemplate
11  10   6  12  15   7   7  12   8  12  0.574903    100/100     NonOverlappingTemplate
8  10  13   9  13   6   7  11  13  10  0.759756    100/100     NonOverlappingTemplate
11  12  10   9  11   9  10  10   9   9  0.999438     97/100     NonOverlappingTemplate
7   6   9   7  11  11  13  13   8  15  0.494392    100/100     NonOverlappingTemplate
11  12  10   8  12  10   8  10   9  10  0.994250     99/100     NonOverlappingTemplate
13  11   7  10   8  16  11  10   5   9  0.474986     99/100     NonOverlappingTemplate
13   6   5  11   5  14  10  13   6  17  0.055361     98/100     NonOverlappingTemplate
11   9   6   8  13  12   8  10  10  13  0.851383    100/100     NonOverlappingTemplate
3   6  13  14  14   9  11  12   9   9  0.249284    100/100     NonOverlappingTemplate
12   6   7  16   7  11   7   8  18   8  0.075719     97/100     NonOverlappingTemplate
10   7  12  12   8   9   8  16  13   5  0.383827     99/100     NonOverlappingTemplate
14  11   6  13   5   9   9  10   7  16  0.249284     98/100     NonOverlappingTemplate
11  11   4  13   9   9   9  17   9   8  0.319084     99/100     NonOverlappingTemplate
13   9   8  11  13   8   7  13  10   8  0.834308     99/100     NonOverlappingTemplate
6   8   5   7  15  15   9  10  11  14  0.202268     99/100     NonOverlappingTemplate
12   8  10  13  11   5   8  15  11   7  0.514124     99/100     NonOverlappingTemplate
9   7  13  10   9  16   9   9   9   9  0.739918     98/100     NonOverlappingTemplate
14   6  11  11   8  10  11   8   8  13  0.779188     98/100     NonOverlappingTemplate
14   8   7   4  12   8  13  15  12   7  0.213309     98/100     NonOverlappingTemplate
5   7  11   6   9  14  12   6  19  11  0.048716    100/100     NonOverlappingTemplate
8  12  10  13   7  12  13   5   9  11  0.678686     98/100     NonOverlappingTemplate
7   9  11   7   6  12  12  17  10   9  0.401199    100/100     NonOverlappingTemplate
7  15  10   9  10  13   7  11   3  15  0.171867    100/100     NonOverlappingTemplate
12  13   5  12  11   6  10  11  12   8  0.657933     99/100     NonOverlappingTemplate
9  11  11  12  13   8   9  12   7   8  0.924076    100/100     NonOverlappingTemplate
6  14  17   7   6  13  15   7   7   8  0.062821    100/100     NonOverlappingTemplate
5   7   7  15  11  12  10   9  10  14  0.437274    100/100     NonOverlappingTemplate
9   6  10   9  12   8  11   8  15  12  0.739918     98/100     NonOverlappingTemplate
12   7   9  10   9   5  10  11  18   9  0.304126     99/100     NonOverlappingTemplate
10  10  11  12  15   6   7  10  10   9  0.779188     99/100     NonOverlappingTemplate
9   8   7   9  13  14  10  12  10   8  0.851383     98/100     NonOverlappingTemplate
11   5  12  11  12  11  10   8   8  12  0.851383    100/100     NonOverlappingTemplate
8  11   5  11   9  16  10   7  10  13  0.474986    100/100     NonOverlappingTemplate
14  10   8   8  11  13  10  12   7   7  0.779188     98/100     NonOverlappingTemplate
14  10   3  16  11   7   8  10  10  11  0.236810     99/100     NonOverlappingTemplate
16   9  11   8   9   7  15   8  12   5  0.275709    100/100     NonOverlappingTemplate
15   8   8   9   9  10  11  11  10   9  0.924076     97/100     NonOverlappingTemplate
9   6   9  17  10  17   9   6   7  10  0.115387     99/100     OverlappingTemplate
12   9  13   7  18  10   8   9   8   6  0.262249     99/100     Universal
9  10  15  12  11   5  10   5  14   9  0.366918     98/100     ApproximateEntropy
4  12   3  15  10   4   7   6   3   7  0.011440     70/71      RandomExcursions
6   8   6   7   8   6   7  11   7   5  0.937294     70/71      RandomExcursions
6   6   4   4   5  12   9   9   8   8  0.491599     71/71      RandomExcursions
2   6   2  10   9  11   9   5   7  10  0.127498     71/71      RandomExcursions
5   7   5   8   9   8   9   4   7   9  0.881013     71/71      RandomExcursions
6   8   4   8  12   8   3   8   8   6  0.519816     69/71      RandomExcursions
5   5   7   5   3   7  10   7   9  13  0.275709     70/71      RandomExcursions
3   6  11   6  14   6   6   3   8   8  0.099089     71/71      RandomExcursions
7  13   8   4   6   4   5   8  10   6  0.339044     71/71      RandomExcursionsVariant
11   9   6   6   8   7   7   1   6  10  0.362174     71/71      RandomExcursionsVariant
10   8   9   8   8   6   8   3   6   5  0.781926     70/71      RandomExcursionsVariant
9   8   7   8   5  10   9   6   6   3  0.754127     71/71      RandomExcursionsVariant
9  10   7   9   5   5   7   9   5   5  0.808725     70/71      RandomExcursionsVariant
5  14   8   5   2   3  11  11   6   6  0.025193     70/71      RandomExcursionsVariant
9   3   6   5  10   8   7   7   8   8  0.808725     70/71      RandomExcursionsVariant
6   5   5   8  10   8   7   9   5   8  0.901761     69/71      RandomExcursionsVariant
3   9   5  12   6   6   6  10   8   6  0.437274     71/71      RandomExcursionsVariant
3   6   7   2  10  11   6  10   7   9  0.238562     71/71      RandomExcursionsVariant
4  10   6  12   3  13   4   4   5  10  0.033552     71/71      RandomExcursionsVariant
7   2  15   7   7   9   7   4   8   5  0.083381     71/71      RandomExcursionsVariant
8   6  13   6   6   6   5   9   7   5  0.577844     71/71      RandomExcursionsVariant
6  15   6   8   6   5   8   4   3  10  0.083381     71/71      RandomExcursionsVariant
11   4   7  10   6  10   3   8   6   6  0.437274     71/71      RandomExcursionsVariant
11   5   4  11   5   9  10   4   7   5  0.295803     69/71      RandomExcursionsVariant
10   5   8   6   4  13   4   7   8   6  0.339044     70/71      RandomExcursionsVariant
10   5   6   7   5  11   8   8   7   4  0.696376     71/71      RandomExcursionsVariant
12   6   8  11  11  15   8   8  11  10  0.739918    100/100     Serial
9   8   6   9  10   6  13  14  15  10  0.455937    100/100     Serial
8   6   7   6  15  10  13  13  14   8  0.289667     97/100     LinearComplexity

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
The minimum pass rate for each statistical test with the exception of the
random excursion (variant) test is approximately = 96 for a
sample size = 100 binary sequences.

The minimum pass rate for the random excursion (variant) test
is approximately = 67 for a sample size = 71 binary sequences.

For further guidelines construct a probability table using the MAPLE program
provided in the addendum section of the documentation.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


I did the same test for the well known cprng salsa20 with same sample size and got very similar results but different p-Values in detail - sometimes higher sometimes lower.

1. Can anybody give me some advice, how to interpret the result?
2. Would it be appropriate to interpret the NIST suite result as "PASSED" even if the results differ from well known pseudo random number generators?
3. Is there another test suite (beside DIEHARD, that does not compile on OSX) to test PRNG random behavior?
• As explained in the answer, the test passed. However, be aware that this is not a good indication (much less proof) that the (p)RNG tested is cryptographically strong; nor that it is correctly implemented. These tests can (only) catch some grossly inadequate RNGs, some implementations errors (hardware or software), and (normal) imperfections in TRNGs. For an illustration of why your three p-values of 0.025, 0.027, and 0.029 are not alarming, there's an obligatory XKCD.
– fgrieu
Oct 28, 2014 at 10:26

The minimum pass rate for each statistical test with the exception of the
random excursion (variant) test is approximately = 96 for a
sample size = 100 binary sequences.

The minimum pass rate for the random excursion (variant) test
is approximately = 67 for a sample size = 71 binary sequences.


This means that every single test can be considered as "passed" if it reports "xx/100" where "xx" is at least 96. The exception is the "RandomExcursions" tests, which report "yy/71" and the threshold is 67.

In your case, all tests pass.

The "P-value" is a synthetic probability; this is what most statistical tests output. In rough terms, when the P-value is (for instance) 0.23, it means that "a perfectly random RNG could have produced a result as skewed as, or more skewed than, what we obtained with probability 0.23". It can be thought of as a measure of implausibility: if the P-value is 0.000001, then this means "we could have obtain such a result from a perfectly fine RNG but it was a one-in-a-million chance, so we don't believe it". See this page for more on this subject.

It is perfectly normal that P-values vary; in fact, if you run the test twice with the same PRNG you will get different values. Moreover, you expect to have a few low P-values, because when you do a hundred tests it is rather normal that you get things which happen only once every 50 times. When running multiple tests, P-values must be "corrected" to account for such an effect (see Bonferroni correction for pointers).

• Thaks a lot. Anyway i wonder about the p-value column. Generally smaller p-values in the table should be better (as usually in statistics)? And shouldn't been have any test rejected that exposes p>0.01?
– ABri
Oct 23, 2014 at 16:16
• Here it is the other way round. Usually, in statistics, you want to detect a non-random effect (e.g. a correlation between a specific gene mutation and a given illness), so you want a small p-value, that would mean "no way this is pure luck, there must be some correlation". Here we really don't want correlations or biases, so we want big p-values. Oct 23, 2014 at 18:23
• The PRNG the NIST-Question referred to is public now: github.com/AndreasBriese/breeze You are a well known cryptographer - maybe you have a look, please.
– ABri
Nov 8, 2014 at 17:40

Here are my answers to your three questions, as well as a final advice in the end.

1. Can anybody give me some advice, how to interpret the result?

Here is an explanation on how to interpret that output data.

In your case, a total of 187 tests (some of the 15 tests actually consist of multiple sub-tests) were conducted to evaluate the randomness of the input data. This number depends on factors such as the template length for the NonOverlappingTemplateMatching test (by default, unless you change some parameters, you will have 187 tests).

The numerous empirical results of these tests were then interpreted with an examination of the proportion of sequences that pass a statistical test (proportion analysis) and the distribution of p-values to check for uniformity (uniformity analysis).

The results of these two analyses are in the output that you posted. Here is how to interpret it.

### Uniformity analysis

The first 10 columns represent the distribution of P-values, and are thus related to the uniformity analysis.

As explained in section 4.2.2 of NIST's paper, the interval between 0 and 1 is divided into 10 sub-intervals, and, after conducting all the iterations of a test, all the P-values that lie within each sub interval are counted and displayed.

In other words, the C1 column contains the number of P-values p for that test such that $0 ≤ p < 0.1$, the C2 column contains the number of P-values p for that test such that $0.1 ≤ p < 0.2$ and so on.

The P-VALUE column contains the P-value of the distribution of the P-values. As explained in the same section of the paper, after counting the number of P-values for each sub-interval, a uniformity analysis is conducted to check if their distribution itself is random.

If it is the case that $P-VALUE ≥ 0.0001$, then the P-values for that test can be considered to be uniformly distributed. If $P-VALUE < 0.0001$, the P-values can’t be considered uniformly distributed, the uniformity analysis is not passed and a * will be shown right next to the P-value.

### Proportion analysis

The PROPORTION column is the count of the sequences that passed the test out of the total number of sequences tested. This ratio is used for the proportion analysis.

As explained in section 4.2.1 of NIST's paper, a confidence interval of acceptable proportions is determined by the test suite. If the proportion falls outside of this interval, then there is evidence that the data is non-random, the proportion analysis is not passed and a * will be shown right next to the proportion.

1. Would it be appropriate to interpret the NIST suite result as "PASSED" even if the results differ from well known pseudo random number generators?

If you interpret the results as explained above, you will safely conclude that your data passed all of the NIST STS tests.

1. Is there another test suite (beside DIEHARD, that does not compile on OSX) to test PRNG random behavior?

You can find some links to other test suites in this answer at Crypto.SE.

• Comments are not for extended discussion; this conversation has been moved to chat. Jan 10, 2017 at 18:19