How do you interpret the p-values from the Dieharder testsuite to evaluate an RNG?

Question

#=============================================================================#
#            dieharder version 3.31.1 Copyright 2003 Robert G. Brown          #
#=============================================================================#
   rng_name    |rands/second|   Seed   |
stdin_input_raw|  3.91e+06  |3556676169|
#=============================================================================#
        test_name   |ntup| tsamples |psamples|  p-value |Assessment
#=============================================================================#
   diehard_birthdays|   0|       100|    1000|0.04228375|  PASSED
      diehard_operm5|   0|   1000000|    1000|0.19715383|  PASSED
  diehard_rank_32x32|   0|     40000|    1000|0.46351097|  PASSED
    diehard_rank_6x8|   0|    100000|    1000|0.95836167|  PASSED
   diehard_bitstream|   0|   2097152|    1000|0.48181370|  PASSED
        diehard_opso|   0|   2097152|    1000|0.30244550|  PASSED
        diehard_oqso|   0|   2097152|    1000|0.94656094|  PASSED
         diehard_dna|   0|   2097152|    1000|0.02404047|  PASSED
diehard_count_1s_str|   0|    256000|    1000|0.45401565|  PASSED
diehard_count_1s_byt|   0|    256000|    1000|0.48456168|  PASSED

Feeding a stream of random numbers to the Dieharder testsuite invokes a list of tests. Each test returns a p-value. As I understand it, the p-value tells you the chance that the stream really is just random noise (or not noise). But I've read that a good RNG will have a range of p-values that follows a uniform distribution; values between 0 and 1 should happen with about equal probability. Why should that be so? Why is a uniform distribution desirable here, what does it mean, and how can you measure the "uniformity" of a distribution?

Answer 1

I've read that a good RNG will have a range of p-values that follows a uniform distribution; values between 0 and 1 should happen with about equal probability. Why should that be so?

It comes straight from the definition of p-values. The p-value indicates the probability you'd get at least that skewed a result if the source is truly random.

So you expect a p-value $\le 0.05$ 5% of the time. You expect a p-value $\le 0.10$ 10% of the time. Thus, you expect a p-value in the interval $[0.05, 0.10]$ 5% of the time. Similarly for any other intervals between zero and one.

Of course, a uniform distribution will not produce an equal number of results in each range, so you will always get a randomly skewed distribution of p-values as well. In practice, you can probably just rely on dieharder to tell you you've passed the tests... keeping in mind that it does not prove you have a good (cryptographic) random number generator.