NIST SP 800-22 on page 1-4 (that's page 16 of the PDF) states the following. Pay attention to the bold passages. (Emphasis are mine.) Are they true?! Can you advise?

One of the primary goals of the following tests is to minimize the probability of a Type II error, i.e., to minimize the probability of accepting a sequence being produced by a generator as good when the generator was actually bad. The probabilities α and β are related to each other and to the size n of the tested sequence in such a way that if two of them are specified, the third value is automatically determined. Practitioners usually select a sample size n and a value for α (the probability of a Type I error – the level of significance). Then a critical point for a given statistic is selected that will produce the smallest β (the probability of a Type II error). That is, a suitable sample size is selected along with an acceptable probability of deciding that a bad generator has produced the sequence when it really is random. Then the cutoff point for acceptability is chosen such that the probability of falsely accepting a sequence as random has the smallest possible value.

In particular in the context of random number generation, if the null hypothesis is false --- that is, if the generator is a bad one ---, there is an infinite number of possibilities for it not being random, so we don't know exactly what distribution of probability to use, so we can't calculate $\beta$.

Now, look at page 16 of the PDF. See the paragraph right above. It says precisely what I just said. It reads

The probability of a Type II error is denoted as β. For the test, β is the probability that the test will indicate that the sequence is random when it is not; that is, a “bad” generator produced a sequence that appears to have random properties. Unlike α, β is not a fixed value. β can take on many different values because there are an infinite number of ways that a data stream can be non-random, and each different way yields a different β. The calculation of the Type II error β is more difficult than the calculation of α because of the many possible types of non-randomness.

So how come they state the bold passages above? Can you educate me on this?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.