Return to Answer

Testing the frequency of bytes produced by an ideal (uniform) random generator per question's method with $1200$ samples or more gives a $\chi^2$ that varies typically by few dozens around the average, $255$. It should be below $254.334$ for 50% of experiments, and the most likely values those closest to $253$.

For a good uniform random sequence of bytes, and "enough" samples, the distribution of $\chi^2$ computed as in the question has a well-studied distribution: the $\chi^2$ distribution with $2^8-1=255$ degrees of freedom, with this Probability Density Function:

The mean is $255$ (that's what you obtain by averaging the $\chi^2$ of many experiments). The median is $\approx254.334$ (the $\chi^2$ is less than this for 50% of experiments). The mode is $253$ (that's the most likely value, maximizing the PDF).

The standard deviation is $\sigma=\sqrt{510}\approx22.58$. Because the PDF is close to a normal, the 68–95–99.7 rule applies: that gives the probability (in percent) to be one, two and three $\sigma$ from the mean. Again that's with "enough" samples. A rule of thumb tells that about $5$ samples per expected value (thus like $1200$ samples) starts to be enough when reasoning at up to $2\sigma$. More sample usually help (I wish I knew how to more rigorously choose the minimum number of samples).

The Cumulative Density Function $\displaystyle\operatorname{CDF}(x)=\int_0^x\operatorname{PDF}(u)\ du$ is:

That tells the expected proportion of experiments (for a uniform random sequence) where the computed $\chi^2$ is less than a certain value.

We have a 0.05% chance that the $\chi^2$ is less than $187.171$, and 0.05% chance that it is above $335.917$. Thus in only about one experiment out of 1000 will the $\chi^2$ will be out of that range, for a truly uniform random sequence, and "enough" samples.

In particular, a generator that consistently gives low $\chi^2$ is not random!

When we generalize to different tests with a random variable taking one out of $n\ge2$ different outcomes with expected frequencies $e_i$ and $\displaystyle1=\sum e_i$, and the number of samples is at least about $5/\min(e_i)$ (enough for a minimum of $5$ samples for each value), then the distribution of $\chi^2$ is the $\chi^2$ distribution with $k=n-1$ degree(s) of liberty.

The shape of the Probability Density Function vary with $k$.

The mean is $k$, the median is about $k(1-2/9k)^3$, the mode is $\max(0, k-2)$.
The variance is $\sigma=\sqrt{2k}$, but for low $k$ computations or tables (rather than estimations based on $\sigma$ and a normal approximation) are necessary.

The shape of the Cumulative Density Function vary with $k$:

Testing the frequency of bytes produced by an ideal (uniform) random generator per question's method with $1200$ samples or more gives a $\chi^2$ that varies typically by few dozens around the average, $255$. It should be below $254.334$ for 50% of experiments, and the most likely values those closest to $253$.

For a good uniform random sequence of bytes, and "enough" samples, the distribution of $\chi^2$ computed as in the question has a well-studied distribution: the $\chi^2$ distribution with $2^8-1=255$ degrees of freedom, with this Probability Density Function:

The mean is $255$ (that's what you obtain by averaging the $\chi^2$ of many experiments). The median is $\approx254.334$ (the $\chi^2$ is less than this for 50% of experiments). The mode is $253$ (that's the most likely value, maximizing the PDF).

The standard deviation is $\sigma=\sqrt{510}\approx22.58$. Because the PDF is close to a normal, the 68–95–99.7 rule applies: that gives the probability (in percent) to be one, two and three $\sigma$ from the mean. Again that's with "enough" samples. A rule of thumb tells that about $5$ samples per expected value (thus like $1200$ samples) starts to be enough when reasoning at up to $2\sigma$. More sample usually help (I wish I knew how to more rigorously choose the minimum number of samples).

The Cumulative Density Function $\displaystyle\operatorname{CDF}(x)=\int_0^x\operatorname{PDF}(u)\ du$ is:

That tells the expected proportion of experiments (for a uniform random sequence) where the computed $\chi^2$ is less than a certain value.

We have a 0.05% chance that the $\chi^2$ is less than $187.171$, and 0.05% chance that it is above $335.917$. Thus in only about one experiment out of 1000 will the $\chi^2$ will be out of that range, for a truly uniform random sequence, and "enough" samples.

In particular, a generator that consistently gives low $\chi^2$ is not random!

When we generalize to different tests with a random variable taking one out of $n\ge2$ different outcomes with expected frequencies $e_i$ and $\displaystyle1=\sum e_i$, and the number of samples is at least about $5/\min(e_i)$ (enough for a minimum of $5$ expected samples for each value), then the distribution of $\chi^2$ is the $\chi^2$ distribution with $k=n-1$ degree(s) of liberty.

The shape of the Probability Density Function vary with $k$.

The mean is $k$, the median is about $k(1-2/9k)^3$, the mode is $\max(0, k-2)$.
The variance is $\sigma=\sqrt{2k}$, but for low $k$ computations or tables (rather than estimations based on $\sigma$ and a normal approximation) are necessary.

The shape of the Cumulative Density Function vary with $k$:

Testing an ideal (uniform) random generator by the method in the question with $1200$ samples or more should give a $\chi^2$ that varies by few dozens around the average, $255$. It should be below $254.334$ for 50% of experiments, and the most likely values those closest to $253$.

For a good uniform random sequence of bytes, and "enough" samples, the distribution of $\chi^2$ computed as in the question has a well-studied distribution: the $\chi^2$ distribution with $2^8-1=255$ degrees of freedom, with this Probability Density Function:

The mean is $255$ (that's what you obtain is you average the $\chi^2$ of many experiments). The median is $\approx254.334$ (the $\chi^2$ is less than this for 50% of experiments). The mode is $253$ (that's the most likely value, maximizing the PDF).

The standard deviation is $\sigma=\sqrt{510}\approx22.58$. Because the PDF is close to a normal, the 68–95–99.7 rule applies: that gives the probability (in percent) to be one, two and three $\sigma$ from the mean. Again that's with "enough" samples. A rule of thumb tells that about $5$ samples per expected value (thus like $1200$ samples) starts to be enough when reasoning at up to $2\sigma$. More sample usually help (I wish I knew how to more rigorously choose the minimum number of samples).

Another way to look at the Cumulative Density Function $\displaystyle\operatorname{CDF}(x)=\int_0^x\operatorname{PDF}(u)\ du$

That tells the expected proportion of experiments (for a uniform random sequence) where the computed $\chi^2$ is less than a certain value.

We have a 0.05% chance that the $\chi^2$ is less than $187.171$, and 0.05% chance that it is above $335.917$. Thus in only about one experiment out of 1000 will the $\chi^2$ will be out of that range, for a truly uniform random sequence, and "enough" samples.

In particular, a generator that consistently give unduly low $\chi^2$ is not random!

When we generalize to different tests with a random variable taking one out of $n\ge2$ different outcomes with expected frequencies $e_i$ and $\displaystyle1=\sum e_i$, and the number of samples is at least about $5/\min(e_i)$ (enough for a minimum of $5$ samples for each value), then the distribution of $\chi^2$ is the $\chi^2$ distribution with $k=n-1$ degree(s) of liberty.

The shape of the Probability Density Function vary with $k$.

The mean is $k$, the median is about $k(1-2/9k)^3$, the mode is $\max(0, k-2)$.
The variance is $\sigma=\sqrt{2k}$, but for low $k$ computations or tables (rather than estimations based on $\sigma$ and a normal approximation) are customary.

The shape of the Cumulative Density Function vary with $k$:

Testing the frequency of bytes produced by an ideal (uniform) random generator per question's method with $1200$ samples or more gives a $\chi^2$ that varies typically by few dozens around the average, $255$. It should be below $254.334$ for 50% of experiments, and the most likely values those closest to $253$.

For a good uniform random sequence of bytes, and "enough" samples, the distribution of $\chi^2$ computed as in the question has a well-studied distribution: the $\chi^2$ distribution with $2^8-1=255$ degrees of freedom, with this Probability Density Function:

The mean is $255$ (that's what you obtain by averaging the $\chi^2$ of many experiments). The median is $\approx254.334$ (the $\chi^2$ is less than this for 50% of experiments). The mode is $253$ (that's the most likely value, maximizing the PDF).

The standard deviation is $\sigma=\sqrt{510}\approx22.58$. Because the PDF is close to a normal, the 68–95–99.7 rule applies: that gives the probability (in percent) to be one, two and three $\sigma$ from the mean. Again that's with "enough" samples. A rule of thumb tells that about $5$ samples per expected value (thus like $1200$ samples) starts to be enough when reasoning at up to $2\sigma$. More sample usually help (I wish I knew how to more rigorously choose the minimum number of samples).

The Cumulative Density Function $\displaystyle\operatorname{CDF}(x)=\int_0^x\operatorname{PDF}(u)\ du$ is:

That tells the expected proportion of experiments (for a uniform random sequence) where the computed $\chi^2$ is less than a certain value.

We have a 0.05% chance that the $\chi^2$ is less than $187.171$, and 0.05% chance that it is above $335.917$. Thus in only about one experiment out of 1000 will the $\chi^2$ will be out of that range, for a truly uniform random sequence, and "enough" samples.

In particular, a generator that consistently gives low $\chi^2$ is not random!

When we generalize to different tests with a random variable taking one out of $n\ge2$ different outcomes with expected frequencies $e_i$ and $\displaystyle1=\sum e_i$, and the number of samples is at least about $5/\min(e_i)$ (enough for a minimum of $5$ samples for each value), then the distribution of $\chi^2$ is the $\chi^2$ distribution with $k=n-1$ degree(s) of liberty.

The shape of the Probability Density Function vary with $k$.

The mean is $k$, the median is about $k(1-2/9k)^3$, the mode is $\max(0, k-2)$.
The variance is $\sigma=\sqrt{2k}$, but for low $k$ computations or tables (rather than estimations based on $\sigma$ and a normal approximation) are necessary.

The shape of the Cumulative Density Function vary with $k$:

Testing an ideal (uniform) random generator by the method in the question with $1200$ samples or more should give a $\chi^2$ that varies by few dozens around the average, $255$. It should be below $254.334$ for 50% of experiments, and the most likely values those closest to $253$.

For a good uniform random sequence of bytes, and "enough" samples, the distribution of $\chi^2$ computed as in the question has a well-studied distribution: the $\chi^2$ distribution with $2^8-1=255$ degrees of freedom, with this Probability Density Function:

The mean is $255$ (that's what you obtain is you average the $\chi^2$ of many experiments). The median is $\approx254.334$ (the $\chi^2$ is less than this for 50% of experiments). The mode is $253$ (that's the most likely value, maximizing the PDF).

The standard deviation is $\sigma=\sqrt{510}\approx22.58$. Because the PDF is close to a normal, the 68–95–99.7 rule applies: that gives the probability (in percent) to be one, two and three $\sigma$ from the mean. Again that's with "enough" samples. A rule of thumb tells that about $5$ samples per expected value (thus like $1200$ samples) starts to be enough when reasoning at up to $2\sigma$. More sample usually help (I wish I knew how to more rigorously choose the minimum number of samples).

Another way to look at the Cumulative Density Function $\displaystyle\operatorname{CDF}(x)=\int_0^x\operatorname{PDF}(y)\ dy$

That tells the expected proportion of experiments (for a uniform random sequence) where the computed $\chi^2$ is less than a certain value.

We have a 0.05% chance that the $\chi^2$ is less than $187.171$, and 0.05% chance that it is above $335.917$. Thus in only about one experiment out of 1000 will the $\chi^2$ will be out of that range, for a truly uniform random sequence, and "enough" samples.

In particular, a generator that consistently give $0$ or other unduly low $\chi^2$ is not random!

When we generalize to different tests with a random variable taking one out of $n\ge2$ different outcomes with expected frequencies $e_i$ and $\displaystyle1=\sum e_i$, and the number of samples is at least about $5/\min(e_i)$ (enough for a minimum of $5$ samples for each value), then the distribution of $\chi^2$ is the $\chi^2$ distribution with $k=n-1$ degree(s) of liberty.

The shape of the Probability Density Function vary with $k$, and for low $k$ is very far from a normal.

The mean is $k$, the median is about $k(1-2/9k)^3$, the mode is $\max(0, k-2)$.
The variance is $\sqrt{2k}$.

The shape of the Cumulative Density Function vary with $k$:

Testing an ideal (uniform) random generator by the method in the question with $1200$ samples or more should give a $\chi^2$ that varies by few dozens around the average, $255$. It should be below $254.334$ for 50% of experiments, and the most likely values those closest to $253$.

For a good uniform random sequence of bytes, and "enough" samples, the distribution of $\chi^2$ computed as in the question has a well-studied distribution: the $\chi^2$ distribution with $2^8-1=255$ degrees of freedom, with this Probability Density Function:

The mean is $255$ (that's what you obtain is you average the $\chi^2$ of many experiments). The median is $\approx254.334$ (the $\chi^2$ is less than this for 50% of experiments). The mode is $253$ (that's the most likely value, maximizing the PDF).

The standard deviation is $\sigma=\sqrt{510}\approx22.58$. Because the PDF is close to a normal, the 68–95–99.7 rule applies: that gives the probability (in percent) to be one, two and three $\sigma$ from the mean. Again that's with "enough" samples. A rule of thumb tells that about $5$ samples per expected value (thus like $1200$ samples) starts to be enough when reasoning at up to $2\sigma$. More sample usually help (I wish I knew how to more rigorously choose the minimum number of samples).

Another way to look at the Cumulative Density Function $\displaystyle\operatorname{CDF}(x)=\int_0^x\operatorname{PDF}(u)\ du$

That tells the expected proportion of experiments (for a uniform random sequence) where the computed $\chi^2$ is less than a certain value.

We have a 0.05% chance that the $\chi^2$ is less than $187.171$, and 0.05% chance that it is above $335.917$. Thus in only about one experiment out of 1000 will the $\chi^2$ will be out of that range, for a truly uniform random sequence, and "enough" samples.

In particular, a generator that consistently give unduly low $\chi^2$ is not random!

When we generalize to different tests with a random variable taking one out of $n\ge2$ different outcomes with expected frequencies $e_i$ and $\displaystyle1=\sum e_i$, and the number of samples is at least about $5/\min(e_i)$ (enough for a minimum of $5$ samples for each value), then the distribution of $\chi^2$ is the $\chi^2$ distribution with $k=n-1$ degree(s) of liberty.

The shape of the Probability Density Function vary with $k$.

The mean is $k$, the median is about $k(1-2/9k)^3$, the mode is $\max(0, k-2)$.
The variance is $\sigma=\sqrt{2k}$, but for low $k$ computations or tables (rather than estimations based on $\sigma$ and a normal approximation) are customary.

The shape of the Cumulative Density Function vary with $k$: