Twofish fails Dieharder test

Question

I've tested Twofish (256 bit) as follows. A lengthy pseudorandom sequence was generated and used as a key. Plaintext phrases 000000000..00, 00000..01, 0000...FFFFF, were generated and encoded using Twofish with the key. Ciphertexts were split in 32 bit numbers (40 million numbers total).

In theory, the cipher should pass the Dieharder tests. In practice, it appears to fail a number of them.

Pseudorandom number generator passed all tests, so this isn't a bug with Dieharder.

Does this mean that Twofish is not as secure as claimed?

---

Edited:

TwoFish.java comes from http://www.sethi.org/tmp/ssh/src/com/mindbright/security/cipher/Twofish.java

FishRunner.java is my wrapper that generates "sequential plain text blocks as ASCII characters". Each block is ciphered using Twofish with byte[] defaultKey = {-24,29,83,38,-77,-92,-46,25,-117,-71,42,-44,-52,97,58,-114,50,-27,49,47,12,52,-76,-88,-26,17,18,84,30,-95,80,13,32};. From each plaintext block I get 256 bits of ciphered text. These 256 bits are split in chunks of 32 bit each and saved to "/tmp/FromFish/fish.txt" with an appropriate header.

first several lines of fish.txt

#==================================================================
# generator mt19937  seed = 316179543
#==================================================================
type: d
count: 40000000
numbit: 32
4205098517
1950644630
 217236044
4021751533
1670623305
4216006007
1982760947
2956150679
2523872870
1802016715
 546984380
3195151793
 478820823
3432394711

Once all 40000000 records were created (440Mb of data) Dieharder was launched with dieharder -a -g 202 -f fish.txt >>res.txt

            dieharder version 3.31.1 Copyright 2003 Robert G. Brown          #
##  
   rng_name    |           filename             |rands/second|
     file_input|                        fish.txt|  7.26e+06  |
#   
        test_name   |ntup| tsamples |psamples|  p-value |Assessment
# b       #
   diehard_birthdays|   0|       100|     100|0.65344498|  PASSED     
      diehard_operm5|   0|   1000000|     100|0.11726095|  PASSED     
  diehard_rank_32x32|   0|     40000|     100|0.84203088|  PASSED     
    diehard_rank_6x8|   0|    100000|     100|0.73796762|  PASSED     
   diehard_bitstream|   0|   2097152|     100|0.38051493|  PASSED      
        diehard_opso|   0|   2097152|     100|0.04995365|  PASSED      
        diehard_oqso|   0|   2097152|     100|0.59990831|  PASSED      
         diehard_dna|   0|   2097152|     100|0.62220146|  PASSED      
diehard_count_1s_str|   0|    256000|     100|0.86262588|  PASSED       
diehard_count_1s_byt|   0|    256000|     100|0.12560712|  PASSED       
 diehard_parking_lot|   0|     12000|     100|0.28041223|  PASSED       
    diehard_2dsphere|   2|      8000|     100|0.76183470|  PASSED       
    diehard_3dsphere|   3|      4000|     100|0.17846568|  PASSED       
     diehard_squeeze|   0|    100000|     100|0.00384865|   WEAK        
        diehard_sums|   0|       100|     100|0.43470008|  PASSED       
        diehard_runs|   0|    100000|     100|0.70560801|  PASSED         
        diehard_runs|   0|    100000|     100|0.98961342|  PASSED       
       diehard_craps|   0|    200000|     100|0.18043071|  PASSED        
       diehard_craps|   0|    200000|     100|0.89290364|  PASSED       
 marsaglia_tsang_gcd|   0|  10000000|     100|0.00000000|  FAILED        
 marsaglia_tsang_gcd|   0|  10000000|     100|0.00000000|  FAILED       
         sts_monobit|   1|    100000|     100|0.72613955|  PASSED        
            sts_runs|   2|    100000|     100|0.26712378|  PASSED        
          sts_serial|   1|    100000|     100|0.94935591|  PASSED        
          sts_serial|   2|    100000|     100|0.93198856|  PASSED       
          sts_serial|   3|    100000|     100|0.99554380|   WEAK          
          sts_serial|   3|    100000|     100|0.72444970|  PASSED         
          sts_serial|   4|    100000|     100|0.19981717|  PASSED          
          sts_serial|   4|    100000|     100|0.06124493|  PASSED        
          sts_serial|   5|    100000|     100|0.87811046|  PASSED       
          sts_serial|   5|    100000|     100|0.09618787|  PASSED        
          sts_serial|   6|    100000|     100|0.87317579|  PASSED       
          sts_serial|   6|    100000|     100|0.99688012|   WEAK         
          sts_serial|   7|    100000|     100|0.50043383|  PASSED        
          sts_serial|   7|    100000|     100|0.61746527|  PASSED       
          sts_serial|   8|    100000|     100|0.97376805|  PASSED        
          sts_serial|   8|    100000|     100|0.94872352|  PASSED        
          sts_serial|   9|    100000|     100|0.35958803|  PASSED        
          sts_serial|   9|    100000|     100|0.39252366|  PASSED        
          sts_serial|  10|    100000|     100|0.40682407|  PASSED       
          sts_serial|  10|    100000|     100|0.69846273|  PASSED      
          sts_serial|  11|    100000|     100|0.14757958|  PASSED      
          sts_serial|  11|    100000|     100|0.85142983|  PASSED     
          sts_serial|  12|    100000|     100|0.33628714|  PASSED   
          sts_serial|  12|    100000|     100|0.80400201|  PASSED       
          sts_serial|  13|    100000|     100|0.50635238|  PASSED     
          sts_serial|  13|    100000|     100|0.04113439|  PASSED         
          sts_serial|  14|    100000|     100|0.48030593|  PASSED    
          sts_serial|  14|    100000|     100|0.83615004|  PASSED   
          sts_serial|  15|    100000|     100|0.85634237|  PASSED    
          sts_serial|  15|    100000|     100|0.86413582|  PASSED     
          sts_serial|  16|    100000|     100|0.81247787|  PASSED    
          sts_serial|  16|    100000|     100|0.62279344|  PASSED   
         rgb_bitdist|   1|    100000|     100|0.10521730|  PASSED    
         rgb_bitdist|   2|    100000|     100|0.36419006|  PASSED   
         rgb_bitdist|   3|    100000|     100|0.50848488|  PASSED   
         rgb_bitdist|   4|    100000|     100|0.91911028|  PASSED   
         rgb_bitdist|   5|    100000|     100|0.68355135|  PASSED   
         rgb_bitdist|   6|    100000|     100|0.07298074|  PASSED   
         rgb_bitdist|   7|    100000|     100|0.99703911|   WEAK    
         rgb_bitdist|   8|    100000|     100|0.50843903|  PASSED   
         rgb_bitdist|   9|    100000|     100|0.51151893|  PASSED   
         rgb_bitdist|  10|    100000|     100|0.40558627|  PASSED   
         rgb_bitdist|  11|    100000|     100|0.27640943|  PASSED   
         rgb_bitdist|  12|    100000|     100|0.64415540|  PASSED   
rgb_minimum_distance|   2|     10000|    1000|0.77930849|  PASSED   
rgb_minimum_distance|   3|     10000|    1000|0.89974521|  PASSED   
rgb_minimum_distance|   4|     10000|    1000|0.52655953|  PASSED   
rgb_minimum_distance|   5|     10000|    1000|0.01029230|  PASSED   
    rgb_permutations|   2|    100000|     100|0.49947588|  PASSED    
    rgb_permutations|   3|    100000|     100|0.48976917|  PASSED   
    rgb_permutations|   4|    100000|     100|0.99435735|  PASSED   
    rgb_permutations|   5|    100000|     100|0.18313717|  PASSED   
      rgb_lagged_sum|   0|   1000000|     100|0.11846988|  PASSED   
      rgb_lagged_sum|   1|   1000000|     100|0.20793259|  PASSED   
      rgb_lagged_sum|   2|   1000000|     100|0.08406645|  PASSED   
      rgb_lagged_sum|   3|   1000000|     100|0.00002903|   WEAK    
      rgb_lagged_sum|   4|   1000000|     100|0.00000000|  FAILED   
      rgb_lagged_sum|   5|   1000000|     100|0.00156593|   WEAK    
      rgb_lagged_sum|   6|   1000000|     100|0.13398055|  PASSED   
      rgb_lagged_sum|   7|   1000000|     100|0.00000071|  FAILED  
      rgb_lagged_sum|   8|   1000000|     100|0.07282751|  PASSED   
      rgb_lagged_sum|   9|   1000000|     100|0.00000653|   WEAK    
      rgb_lagged_sum|  10|   1000000|     100|0.24179580|  PASSED   
      rgb_lagged_sum|  11|   1000000|     100|0.00620851|  PASSED   
      rgb_lagged_sum|  12|   1000000|     100|0.24310357|  PASSED   
      rgb_lagged_sum|  13|   1000000|     100|0.15323532|  PASSED   
      rgb_lagged_sum|  14|   1000000|     100|0.00000174|   WEAK    
      rgb_lagged_sum|  15|   1000000|     100|0.00000000|  FAILED   
      rgb_lagged_sum|  16|   1000000|     100|0.39385345|  PASSED   
      rgb_lagged_sum|  17|   1000000|     100|0.11381934|  PASSED   
      rgb_lagged_sum|  18|   1000000|     100|0.78715021|  PASSED   
      rgb_lagged_sum|  19|   1000000|     100|0.00000000|  FAILED   
      rgb_lagged_sum|  20|   1000000|     100|0.52371093|  PASSED   
      rgb_lagged_sum|  21|   1000000|     100|0.00531931|  PASSED   
      rgb_lagged_sum|  22|   1000000|     100|0.19857791|  PASSED   
      rgb_lagged_sum|  23|   1000000|     100|0.00000000|  FAILED   
      rgb_lagged_sum|  24|   1000000|     100|0.00000431|   WEAK    
      rgb_lagged_sum|  25|   1000000|     100|0.02471464|  PASSED   
      rgb_lagged_sum|  26|   1000000|     100|0.07546643|  PASSED   
      rgb_lagged_sum|  27|   1000000|     100|0.01786362|  PASSED   
      rgb_lagged_sum|  28|   1000000|     100|0.28778610|  PASSED   
      rgb_lagged_sum|  23|   1000000|     100|0.00000000|  FAILED 
      rgb_lagged_sum|  30|   1000000|     100|0.13632978|  PASSED   
      rgb_lagged_sum|  31|   1000000|     100|0.00000000|  FAILED   
      rgb_lagged_sum|  32|   1000000|     100|0.11176539|  PASSED   
     rgb_kstest_test|   0|     10000|    1000|0.29113864|  PASSED   
     dab_bytedistrib|   0|  51200000|       1|0.00000000|  FAILED   
             dab_dct| 256|     50000|       1|0.93719430|  PASSED   
Preparing to run test 207.  ntuple = 0 
        dab_filltree|  32|  15000000|        1|0.60930301|  PASSED  
        dab_filltree|  32|  15000000|       1|0.29252232|  PASSED     
Preparing to run test 208.  ntuple = 0   
       dab_filltree2|   0|   5000000|       1|0.64246455|  PASSED     
       dab_filltree2|   1|   5000000|       1|0.20829118|  PASSED     
Preparing to run test 209.  ntuple = 0
        dab_monobit2|  12|  65000000|       1|1.00000000|  FAILED     

Here's a summary of the tests that failed:

       test_name   |ntup| tsamples |psamples|  p-value |Assessment
marsaglia_tsang_gcd|   0|  10000000|     100|0.00000000|  FAILED         
marsaglia_tsang_gcd|   0|  10000000|     100|0.00000000|  FAILED
     rgb_lagged_sum|   7|   1000000|     100|0.00000071|  FAILED
     rgb_lagged_sum|  15|   1000000|     100|0.00000000|  FAILED
     rgb_lagged_sum|  19|   1000000|     100|0.00000000|  FAILED
     rgb_lagged_sum|  23|   1000000|     100|0.00000000|  FAILED
     rgb_lagged_sum|  31|   1000000|     100|0.00000000|  FAILED 
    dab_bytedistrib|   0|  51200000|       1|0.00000000|  FAILED
       dab_monobit2|  12|  65000000|       1|1.00000000|  FAILED

Twofish algo is saved here: https://pastebin.com/THGegxKY

Class below is my wrapper to generate ciphers from consequential plain text messages

package models;


import java.io.File;
import java.io.FileOutputStream;
import java.io.PrintWriter;
import java.util.Arrays;
import java.util.Random;

public class FishRunner {
    
    private  String filePath = "~/Dev/CleanFish/res/";
    private  String valuesFolder = "values/in_";    
    private  String keysFolder = "keys/k_";
    private  String resultsFolder = "results/dh_";
    
    
    
    private int keySize = 32;
    private byte[] defaultKey = {-24,29,83,38,-77,-92,-46,25,-117,-71,42,-44,-52,97,58,-114,50,-27,49,47,12,52,-76,-88,-26,17,18,84,30,-95,80,13};
    
    
    public static void main(String[] args) {
        
        System.out.println("l: " + fS(toUnsignedLong(-1)) +", 2xMaxInt "+ (2* ((long) Integer.MAX_VALUE))  +"\n\n\n");  
        FishRunner fr = new FishRunner();       
    }
        
    private FishRunner() {
        
        int[] numberOfReps = {6180339, 9227465, 10000000, 14930352};  
        char[] codesN = {'a', 'b', 'c','d'};

        byte[] defPlainTextByteValue = {0,1,2,8,15}; 
        char[] codesP = {'0','1','2','8','F'};
        
        byte[] defaultOffsetStartingPosition = {0,3,7,12}; 
        String[] codesOff = {"00","03","07","12"};
        
        
        
        for(int i=0; i< numberOfReps.length; i++) {

            for(int j=0; j< defPlainTextByteValue.length; j++) {

                for(int k=0; k< defaultOffsetStartingPosition.length; k++) {
                    
                    String fName = codesN[i]+ "_"+codesP[j]+ "_"+codesOff[k]+".txt";
                    
                    try {
                        testAlgo(fName, numberOfReps[i], 0, defPlainTextByteValue[j], defaultOffsetStartingPosition[k]) ;
                        System.out.println("Done: "+fName);                 
                    } catch (Exception e) { e.printStackTrace();    }
                    
                }                   
            }
        }
        System.out.println("Done.");
    }
    private void testAlgo(String filename, int numOfRepetitions, int initPlainTextValue, byte backgroundValue, byte defaultOffsetStartingPosition) throws  Exception {

        writeDHCommand(filename);   
        byte[] keyChars = defaultKey;
        
        Object key = TwoFish.makeKey(keyChars);     
        int lineCount = numOfRepetitions*4;
        
        writeKeyToFile(filename,keyChars);  

        
        
        
        PrintWriter writer = new PrintWriter(filePath + valuesFolder + filename, "UTF-8");
        writer.println("#==================================================================");
        writer.println("# generator mt19937  seed = 316179543");
        writer.println("#==================================================================");
        writer.println("type: d");
        writer.println("count: "+lineCount+"");
        writer.println("numbit: 32");
        
        for(int i = 0; i < numOfRepetitions; i++) {         
            int intOffset = i + initPlainTextValue; 
            byte[] text = generateText(intOffset,backgroundValue,defaultOffsetStartingPosition);
             
                byte[] ct = TwoFish.blockEncrypt(text, 0, key);
                for (int x = 0; x < ct.length ; x=x+4) {
                    writer.println(fS(Long.toString(toUnsignedLong(toInt(ct, x))),10));                 
                }
        }
        writer.close(); 
    }


    private byte[] generateText(int stepNumber, byte defValue, byte defaultOffsetStartingPosition) {
        
        byte[] text = new byte[32];
        Arrays.fill(text, defValue);        

        text[(defaultOffsetStartingPosition)% 32] = (byte) (stepNumber );
        text[(defaultOffsetStartingPosition+1)% 32] = (byte) (stepNumber >>> 8);
        text[(defaultOffsetStartingPosition+2)% 32] = (byte) (stepNumber >>> 16);
        text[(defaultOffsetStartingPosition+3)% 32] = (byte) (stepNumber >>> 24);   
        return text;            
        }


    private byte[] initKey() {
        
        Random r = new Random(0);
        byte[] key = new byte[keySize];
        
        for (int i = 0; i <  keySize; i++) {            
            key[i] = (byte)(-128+r.nextInt(256)) ;  
        }
        System.out.println("Key: "+Arrays.toString(key));

    return key;
        
    }   

    public static int toInt(byte[] bytes, int offset) {
        int ret = 0;
        for (int i=0; i<4 && i+offset<bytes.length; i++) {
            ret <<= 8;
            ret |= (int)bytes[i+offset] & 0xFF;
            }
        return ret;
        }
    
    
    public static long toUnsignedLong(int x) {
        return x & 0x00000000ffffffffL;
    }
    
    private static final char[] HEX_DIGITS = {                '0','1','2','3','4','5','6','7','8','9','A','B','C','D','E','F'              };
    
    private static String toString (byte[] ba, int offset, int length) {
        char[] buf = new char[length * 2];
        for (int i = offset, j = 0, k; i < offset+length; ) {
            k = ba[i++];
            buf[j++] = HEX_DIGITS[(k >>> 4) & 0x0F];
            buf[j++] = HEX_DIGITS[ k      & 0x0F];
            }
        return new String(buf);
        } 
    
    private void writeDHCommand(String fileName) throws Exception { 
        
        String formattedFileName =  valuesFolder + fileName;
        String dhFileName = resultsFolder + fileName;
        
        PrintWriter writerDHCommand = new PrintWriter(new FileOutputStream(new File(filePath+"dh_commands.txt"), true));        
        String dhCmdLine = "dieharder -a -g 202 -f "+ formattedFileName+ " >>"+ dhFileName;
        writerDHCommand.println(dhCmdLine);
        writerDHCommand.close();        
    }
    
    private void writeKeyToFile(String filename, byte[] keyChars) throws Exception {    
        PrintWriter keyWritter = new PrintWriter(filePath + keysFolder + filename, "UTF-8");        
        keyWritter.println(Arrays.toString(keyChars));
        keyWritter.close();
    }
    
    
       
       
       //        INNER METHODS
       
    
    private static final int longSpaces = 35;
    private static final int intSpaces = 15;
        
        

        public static String fS(long text) {    
            return fS(Long.toString(text), longSpaces);
        }
        
        public static String fS(int text) { 
            return fS(Integer.toString(text), intSpaces);
        }
        
        public static String fS(String text, int charTotal) {       
            return String.format("%1$" + charTotal + "s",text + "" );
        }
        
        
           private static String toString (byte[] ba) {
                  return toString(ba, 0, ba.length);
               }
}

---

Edited:

Rewinds are not spotted by in 10M x 32bit datasets from Pi Hex and SecureRandom in at least some data sets. Increasing the size of PT for TwoFish allows it to pass DH tests. Possible explainations:

• False negative. The larger the number of sequencial contributes to

Randomness of plaintext blocks increases with their number.

As the number of PT blocks increases

• False positive. Looping over a small data set results in FAILED tests(Unlikely, since Hex Pi passes tests at the same size)
• False negative. Large number of PT blocks increases the entropy of PT itself, allowing the cipher to PASS tests

Passing extensive randomness testing suggests (but doesn't guarantee) that the cipher might be strong. Failing randomness tests indicate a problem with one of the following:

• Weak key (ruled out by testing a set of keys, generated using SecureRandom)
• Wrong conversion from CT to DH input format (if so - what went wrong?)
• Wrong implementation of the cipher (Does Google return buggy TF.java ?)
• Wrong cipher design (unlikely: algo validation improved since 1960s RANDU)

Answer 1

When would-be pseudorandom data fails a randomness test, the reasons are of the following kinds (from most to least common in my experience)

1. The data is generated by a method not supposed to yield pseudorandom data.
2. The test was incorrectly applied.
3. A strike of bad luck: randomness tests are supposed to fail when given truly random data, with probability governed by the acceptance level(s) for their p-value. If neither p nor 1-p are so infinitesimally low as to exclude bad luck, it is useful to re-run the test with a different seed for the data tested.
4. The code that generated the data tested is broken.
5. The test or its implementation is broken.
6. The method to generate the data tested is broken. A few minutes of review of that method is often (and always in my experience) a much less unreliable way to reach that conclusion than running a test is.

---

Here the problem is of kind 2: The tests that fail badly (reported p-value <0.0001 or >0.9999) require more input than is available, by design DieHarder cycles on the input provided, the tests detects redundancy and fail for that reason, as suggested by Paul Uszak's comment. There's a mere file_input was rewound n times warning. From the man page:

Note well: many tests with default parameters require a lot of rands!
(..)
A file that is too small will "rewind" and render the test results where a rewind occurs suspect.
(..)
Note well that file input rands are delivered to the tests on demand, but if the test needs more than are available it simply rewinds the file and cycles through it again, and again, and again as needed. Obviously this significantly reduces the sample space and can lead to completely incorrect results for the p-value histograms unless there are enough rands to run EACH test without repetition (it is harmless to reuse the sequence for different tests). Let the user beware!

I could not find a statement of how large the file needs to be to avoid that circular reuse. I threw 1<<27 where there was 10000000 in the original question's code, for a file with $2^{34}$ random bits diluted as nearly 6GByte of decimal. That removed all failures, but my understanding is that some tests still have looped as much as 7 times over the data. Over the 117 tests, there remain two WEAK with p-value ≈0.00386 and ≈0.99885 (rgb_minimum_distance 4 and rgb_lagged_sum 18), which is not alarming.

Kind 6 could be summarily ruled out with BlowFish enciphering an incremental value: for any passable 128-bit block cipher that can be used to generate pseudorandom data (before about 2⁶⁸ bytes generated, the fact that no two output blocks are equal is not reliably detectable). Don't even think about validating a block cipher design with any ready-made randomness test. Even validating a block cipher implementation gives dubious insurance when the test pass.

---

Before reaching that conclusion, I thought the problem was of kind 1, doubting that the stated procedure would generate pseudorandom data (it does), based on the question's

A lengthy pseudorandom sequence was generated and used as a key. Plaintext phrases 000000000..00, 00000..01, 0000...FFFFF, were generated and encoded using Twofish with the key. Ciphertexts were split in 32 bit numbers (40 million numbers total).

Apparent problem was: when reusing a key, and using no specified encryption mode or no Initialization Vector, it is entirely possible that relatedness of plaintexts (as we have here) leads to relatedness of ciphertexts. For example, encryption of a counter with CTR, CFB or OFB mode, and a fixed/absent IV, leads to badly related ciphertexts. But after diving in the code and its linked pastebin, it enciphers incremental blocks in ECB mode, which (for a 128-bit block cipher at least) is fine.

---

Note: The following is updated for the code in version 6 of the question, which improves performance over version 5 by moving makeKey out of the largest loop, and builds 80 test files instead of 1.

Except for the insufficient output size, the question's code does not cause the issues observed. However it has consistency and readability issues:

• One of of the 80 files produced is for plaintext matching the question's first paragraph, but the example output and DieHarder results are for another one where the counter is at a different position. We do not have DieHarder results for the other files.
• The largest files produced contain 59721408 32-bit numbers, and that's still not enough, I guess.
• The plaintext text is a byte[32] with the last 16 bytes unused.
• The decimal numbers fed out to DieHarder are words of the TwoFish cipher after a mild obfuscation: e.g. 4205098517 (the first value in the sample output) is produced by the following litany of transformations which had to be scrutinized:
  _{At the end of blockEncrypt, the TwoFish 32-bit word held in variable x2 is 364291322 (15B6A4FAh), becomes four byte (little endian, per blowfish's specification: FAh, A4h, B6h, 15h), is back to 32-bit (big-endian!) as FAA4B615h in toInt, goes 64-bit in toUnsignedLong (which actually returns a non-negative signed long), then variable size decimal string 4205098517 in Long.toString, is appended a constant empty string on the right, then converted to 10-character string with (zero) leading space(s) on the left in one of the many variants of fS using String.format with a dynamically constructed format string, then goes from 2-byte characters to UTF-8 with platform-dependent newlines, which is the same as ASCII.}
• Dead code (commented out or otherwise never invoked), many System.out.println intended for debug, and the unrelated # generator mt19937 seed = 316179543 are left in something posted for review.

Recommendation: DieHarder requires so much input that a generator written as an independent program is best tested with pipe mode, similar to:
cat /dev/urandom | dieharder -a -g 200
I guess (not tested) that Java can pipe out with BufferedOutputStream.write. If we want to go thru files, I recommend binary mode -g 201 (>63% smaller); or -g 202 with hexadecimal (still >18% smaller) which is also easier to produce in Java than non-negative decimal (note: after type: x, the count and numbit values remain decimal).

Answer 2

In light of the rewind problem, it might be useful to do a run of unlimited length against /dev/urandom which didn't fail any tests. What follows is the output of dieharder free running and sucking up as much data as it wants:-

$ dd if=/dev/urandom  | pv | dieharder -g 200 -a
#=============================================================================#]
#            dieharder version 3.31.1 Copyright 2003 Robert G. Brown          #
#=============================================================================#
   rng_name    |rands/second|   Seed   |
stdin_input_raw|  3.23e+07  |3814747282|
#=============================================================================#
        test_name   |ntup| tsamples |psamples|  p-value |Assessment
#=============================================================================#
   diehard_birthdays|   0|       100|     100|0.83214290|  PASSED  
      diehard_operm5|   0|   1000000|     100|0.60013256|  PASSED              ]
  diehard_rank_32x32|   0|     40000|     100|0.98277309|  PASSED              ]
    diehard_rank_6x8|   0|    100000|     100|0.93979458|  PASSED              ]
   diehard_bitstream|   0|   2097152|     100|0.23792128|  PASSED              ]
        diehard_opso|   0|   2097152|     100|0.82509531|  PASSED              ]
        diehard_oqso|   0|   2097152|     100|0.99035615|  PASSED              ]
         diehard_dna|   0|   2097152|     100|0.68998287|  PASSED    <=>       ]
diehard_count_1s_str|   0|    256000|     100|0.47344847|  PASSED  
diehard_count_1s_byt|   0|    256000|     100|0.72423351|  PASSED  >           ]
 diehard_parking_lot|   0|     12000|     100|0.33862926|  PASSED              ]
    diehard_2dsphere|   2|      8000|     100|0.59658397|  PASSED  
    diehard_3dsphere|   3|      4000|     100|0.04881531|  PASSED              ]
     diehard_squeeze|   0|    100000|     100|0.50560983|  PASSED              ]
        diehard_sums|   0|       100|     100|0.00511400|  PASSED  
        diehard_runs|   0|    100000|     100|0.92603458|  PASSED              ]
        diehard_runs|   0|    100000|     100|0.79936926|  PASSED  
       diehard_craps|   0|    200000|     100|0.62843174|  PASSED              ]
       diehard_craps|   0|    200000|     100|0.66389176|  PASSED  
 marsaglia_tsang_gcd|   0|  10000000|     100|0.93021321|  PASSED           <=>]
 marsaglia_tsang_gcd|   0|  10000000|     100|0.26275640|  PASSED  
         sts_monobit|   1|    100000|     100|0.73426739|  PASSED  
            sts_runs|   2|    100000|     100|0.93691012|  PASSED   <=>        ]
          sts_serial|   1|    100000|     100|0.91368906|  PASSED              ]
          sts_serial|   2|    100000|     100|0.69849075|  PASSED  
          sts_serial|   3|    100000|     100|0.67460194|  PASSED  
          sts_serial|   3|    100000|     100|0.98646955|  PASSED  
          sts_serial|   4|    100000|     100|0.84715559|  PASSED  
          sts_serial|   4|    100000|     100|0.22658512|  PASSED  
          sts_serial|   5|    100000|     100|0.93542842|  PASSED  
          sts_serial|   5|    100000|     100|0.42186099|  PASSED  
          sts_serial|   6|    100000|     100|0.23292991|  PASSED  
          sts_serial|   6|    100000|     100|0.10652986|  PASSED  
          sts_serial|   7|    100000|     100|0.04661608|  PASSED  
          sts_serial|   7|    100000|     100|0.90747243|  PASSED  
          sts_serial|   8|    100000|     100|0.19517131|  PASSED  
          sts_serial|   8|    100000|     100|0.76979629|  PASSED  
          sts_serial|   9|    100000|     100|0.17477898|  PASSED  
          sts_serial|   9|    100000|     100|0.91722005|  PASSED  
          sts_serial|  10|    100000|     100|0.78098352|  PASSED  
          sts_serial|  10|    100000|     100|0.86615492|  PASSED  
          sts_serial|  11|    100000|     100|0.64873253|  PASSED  
          sts_serial|  11|    100000|     100|0.57981945|  PASSED  
          sts_serial|  12|    100000|     100|0.47634540|  PASSED  
          sts_serial|  12|    100000|     100|0.18449207|  PASSED  
          sts_serial|  13|    100000|     100|0.75645866|  PASSED  
          sts_serial|  13|    100000|     100|0.03539615|  PASSED  
          sts_serial|  14|    100000|     100|0.86965703|  PASSED  
          sts_serial|  14|    100000|     100|0.64070570|  PASSED  
          sts_serial|  15|    100000|     100|0.86951991|  PASSED  
          sts_serial|  15|    100000|     100|0.81943081|  PASSED  
          sts_serial|  16|    100000|     100|0.57088163|  PASSED  
          sts_serial|  16|    100000|     100|0.29040060|  PASSED  
         rgb_bitdist|   1|    100000|     100|0.86873833|  PASSED              ]
         rgb_bitdist|   2|    100000|     100|0.85325236|  PASSED              ]
         rgb_bitdist|   3|    100000|     100|0.95226081|  PASSED              ]
         rgb_bitdist|   4|    100000|     100|0.66038635|  PASSED              ]
         rgb_bitdist|   5|    100000|     100|0.13784122|  PASSED              ]
         rgb_bitdist|   6|    100000|     100|0.27142406|  PASSED              ]
         rgb_bitdist|   7|    100000|     100|0.61172905|  PASSED              ]
         rgb_bitdist|   8|    100000|     100|0.39661910|  PASSED              ]
         rgb_bitdist|   9|    100000|     100|0.45469854|  PASSED              ]
         rgb_bitdist|  10|    100000|     100|0.91797334|  PASSED   <=>        ]
         rgb_bitdist|  11|    100000|     100|0.98811206|  PASSED              ]
         rgb_bitdist|  12|    100000|     100|0.24174300|  PASSED              ]
rgb_minimum_distance|   2|     10000|    1000|0.53009831|  PASSED              ]
rgb_minimum_distance|   3|     10000|    1000|0.99123568|  PASSED  >           ]
rgb_minimum_distance|   4|     10000|    1000|0.05278055|  PASSED           <=>]
rgb_minimum_distance|   5|     10000|    1000|0.00192692|   WEAK               ]
    rgb_permutations|   2|    100000|     100|0.25534804|  PASSED              ]
    rgb_permutations|   3|    100000|     100|0.95871871|  PASSED              ]
    rgb_permutations|   4|    100000|     100|0.69024074|  PASSED              ]
    rgb_permutations|   5|    100000|     100|0.62811376|  PASSED              ]
      rgb_lagged_sum|   0|   1000000|     100|0.43118771|  PASSED              ]
      rgb_lagged_sum|   1|   1000000|     100|0.78947942|  PASSED              ]
      rgb_lagged_sum|   2|   1000000|     100|0.54082929|  PASSED              ]
      rgb_lagged_sum|   3|   1000000|     100|0.89304502|  PASSED              ]
      rgb_lagged_sum|   4|   1000000|     100|0.54922923|  PASSED              ]
      rgb_lagged_sum|   5|   1000000|     100|0.52252702|  PASSED   <=>        ]
      rgb_lagged_sum|   6|   1000000|     100|0.50123263|  PASSED              ]
      rgb_lagged_sum|   7|   1000000|     100|0.95607041|  PASSED              ]
      rgb_lagged_sum|   8|   1000000|     100|0.78358255|  PASSED              ]
      rgb_lagged_sum|   9|   1000000|     100|0.93639924|  PASSED        <=>   ]
      rgb_lagged_sum|  10|   1000000|     100|0.96720351|  PASSED              ]
      rgb_lagged_sum|  11|   1000000|     100|0.76648842|  PASSED              ]
      rgb_lagged_sum|  12|   1000000|     100|0.95844403|  PASSED  <=>         ]
      rgb_lagged_sum|  13|   1000000|     100|0.99628444|   WEAK               ]
      rgb_lagged_sum|  14|   1000000|     100|0.17004949|  PASSED         <=>  ]
      rgb_lagged_sum|  15|   1000000|     100|0.78382985|  PASSED              ]
      rgb_lagged_sum|  16|   1000000|     100|0.07884512|  PASSED           <=>]
      rgb_lagged_sum|  17|   1000000|     100|0.23564181|  PASSED              ]
      rgb_lagged_sum|  18|   1000000|     100|0.99832213|   WEAK               ]
      rgb_lagged_sum|  19|   1000000|     100|0.08803558|  PASSED              ]
      rgb_lagged_sum|  20|   1000000|     100|0.92950233|  PASSED              ]
      rgb_lagged_sum|  21|   1000000|     100|0.21138751|  PASSED  <=>         ]
      rgb_lagged_sum|  22|   1000000|     100|0.33451827|  PASSED              ]
      rgb_lagged_sum|  23|   1000000|     100|0.80858319|  PASSED              ]
      rgb_lagged_sum|  24|   1000000|     100|0.51121780|  PASSED              ]
      rgb_lagged_sum|  25|   1000000|     100|0.53501596|  PASSED  >           ]
      rgb_lagged_sum|  26|   1000000|     100|0.10784878|  PASSED     <=>      ]
      rgb_lagged_sum|  27|   1000000|     100|0.83026194|  PASSED              ]
      rgb_lagged_sum|  28|   1000000|     100|0.67926963|  PASSED              ]
      rgb_lagged_sum|  29|   1000000|     100|0.90806411|  PASSED              ]
      rgb_lagged_sum|  30|   1000000|     100|0.98452920|  PASSED              ]
      rgb_lagged_sum|  31|   1000000|     100|0.51741868|  PASSED              ]
      rgb_lagged_sum|  32|   1000000|     100|0.79835395|  PASSED              ]
     rgb_kstest_test|   0|     10000|    1000|0.01515269|  PASSED  
     dab_bytedistrib|   0|  51200000|       1|0.33023025|  PASSED              ]
             dab_dct| 256|     50000|       1|0.08974740|  PASSED              ]
Preparing to run test 207.  ntuple = 0
        dab_filltree|  32|  15000000|       1|0.28064178|  PASSED              ]
        dab_filltree|  32|  15000000|       1|0.85469896|  PASSED  
Preparing to run test 208.  ntuple = 0
       dab_filltree2|   0|   5000000|       1|0.09419009|  PASSED              ]
       dab_filltree2|   1|   5000000|       1|0.02769773|  PASSED  
Preparing to run test 209.  ntuple = 0
        dab_monobit2|  12|  65000000|       1|0.30580656|  PASSED              ]
 229GiB 0:37:04 [ 105MiB/s] [                       <=>                        ]

which took 37 minutes. Notice the generation rate of 3.23e+07 rands /second. A rand is a 4 byte word, so the tests ran at 129 MB/s for 37 minutes. The final line shows a total byte consumption of 229GiB which is a surprising 246GB. Compare this to the 160Mb used in the question.

Not evident from the listing, is that the dieharder diehard tests themselves consumed ~5GB of randomness. The original diehard tests only require 10MB.

This then raises the question of whether a dieharder test on anything short of ~250GB is entirely reliable, considering the warning about the effect of rewinds. Robert Brown must have had some idea of the recommended sample size other than "modern random number generators in a typical simulation application can easily need to generate 10^18 or more random numbers". It would be useful to see such a recommendation as we have for ent and diehard's ancestor, diehard.

A further implication is that dieharder is not the test of choice for DIY entropists. There are a number of DIY TRNGs on this site using everything from radiation, through uninitialised RAM states to diodes and webcams. dieharder's seemingly obfuscated (but large) sample size requirement precludes it's use to reliably test these designs..

Answer 3

There is another possible hypothesis...

The tests are buggy. That's an alternative hypothesis acknowledged in the dieharder documentations, as:-

Even small errors in test statistics permit the alternative (usually unstated) null hypothesis to become an important factor in rng testing -- the unwelcome possibility that your generator is just fine but it is the test that is failing.

A simple example is generator 207 XOR (supergenerator). That's an XOR of AES, threefish and kiss. I just get Segmentation fault (core dumped)! How?

But now the more complex examples. This is a snippet from a test of 100MB of /dev/urandom:-

marsaglia_tsang_gcd|   0|  10000000|     100|0.00000000|  FAILED

Look at the p value. It cannot be exactly 0 on a test run over ChaCha20. It can be outside of dieharder's default 0.005 threshold, but not zero. You've got zero's too for that same test. It's statistically impossible to have as many p=0s as you have. I only had one FAILED, but my version is newer probably.

And unfortunately, this also extends to NIST's statistical test suite. This is one line from a run over the same 100MB file:-

Notice that p=0.000001. But, look closer. The sum of the passing p values is 10 (add them up), yet the test reports 9/10! That may have contributed to the p=0 calculation. And that single report has another 19 such instances, mainly in the overlappingtemplate tests, but also in the rank test. How?

This suggests that for randomness testing, $P(H_1) \gg 0$.

---

I've added another answer as the other one is just a demonstrative unconstrained run.