0
$\begingroup$

I've generated 100M of random numbers in range (0..255). These numbers fail Dieharder tasts (bitnum = 8). However, I can pass this test by combining four bytes into an int. Randomness of the set didn't change, so, something isn't correct with the test settings. What could cause it?

    def prep_input(file_name, number_of_iterations,numbit=32):
        f = open(file_name, "a")            
        max_rand = (-1 + 2 ** numbit)
        print(str(max_rand))
        a = len(str(max_rand))
        for i in range(0, number_of_iterations):
            f.write(str(randint(0, max_rand)).rjust(a) + "\n")

        f1.close()

With header it looks like

#==================================================================
# Dieharder
#==================================================================
type: d
count: 16180337
numbit: 8
255
  0
 93
 ....

dieharder -a -g 202 -f input.txt >>result.txt

Report on bytes: Set fails when bytes are used
Report on ints: Set passes tests when ints are used


Update:

  • I've tested up to 4Gb of data. This is close to 2^32 although ASCII format requires ~4 characters (4 bytes) per a byte of actual data.
  • When this file is converted to byte stream it passws several tests and then reaches the end of data. When concatinated it passes tests better than ASCII int version.
  • When Python random is fed as a binary to

    import time import sys import os from random import randint

    j=0 div = 2

    while(True): sys.stdout.write(chr(randint(0, 255))) # newFileBytes = [randint(0, 255)] # os.write(1, (''.join(chr(i) for i in newFileBytes)).encode('charmap')) sys.stdout.flush() j+=1

    if 0 == j%div :
        dh_file = open("progress.log", "a")
        dh_file.write(str(j)+"\n")
        dh_file.close()
        div = div*2
    

With python to_terminal.py | dieharder -a -g 200 >>test_result.dhres I get 16G (17179869184) as the last record in the log and out of 75 tests only two tests are weak the rest is a pass. Python passes DH when supplied as bit stream Clearly, the problem follows 8 bit ASCII representation.

$\endgroup$
  • $\begingroup$ Randomness is fundamentally impossible to measure. All statistical tests simply estimate randomness in different ways. I don't think it's surprising that different statistical tests produce different results. (Concretely, the output does look like something may be wrong, though.) $\endgroup$ – yyyyyyy Jan 20 at 23:29
  • $\begingroup$ Standard PRNG, 90% increase in pass rate thanks to data aggregation? There must be something wrong with the DH run parameters. $\endgroup$ – Stepan Jan 21 at 1:18
  • $\begingroup$ Feel free to comment downvotes. $\endgroup$ – Stepan Jan 21 at 1:22
  • $\begingroup$ Run the basic ENT test across your bytified data. If it passes, you've done the Dieharder test wrong. If it fails ENT badly, you've converted the data wrong. Then take it from there... $\endgroup$ – Paul Uszak Jan 21 at 4:19
  • 1
    $\begingroup$ dieharder -a requires much more data (in the order of $2^{32}$ bytes), see citations from the manual there. Also, testing a would-be CS(P)RNG using Diehard is somewhat like testing a bathyscaphe in a pool: not detecting a failure allows to draw no operational conclusion. $\endgroup$ – fgrieu Jan 21 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.