My question is whether compressing the generated file is a proper test? Does the algorithm fail because of the generated file can be compressed?
You've misinterpreted a small but crucial step in your randomness test. It is perfectly acceptable to output encoded data for randomness testing via compression. HEX, ASCII or plain Klingon is fine. It's a good litmus test without resorting to further exotic testing. Your compression calculation was wrong.
When you output a 100K sequence, that must mean you output 100K samples, not digits. If your generator is RC4, you will be outputting bytes that might be represented by up to three characters. If you're using Java's Random() you will be outputting 32 bit values that might be 10 characters long. There might also be spaces, commas, carriage returns and newlines. These latter artefacts will effectively create what's termed a bit fixing entropy source. In extremis, you might print "It's a bit!" and "I got nought" for each Blum Blum Shub (BBS) output. No matter.
- Output 100K samples, with any encoding.
- This might occupy 1200K storage bytes as ASCII representations of bits from BBS using my special encoding above.
- Compress with the best squeezer. fp8 is better than 7z which is better than zip.
- Divide the resulting compressed size by 100K to get the entropy rate/ sample.
If your secret algorithm is any good, you'll get an entropy rate approaching 1 bit /sample which is equivalent to 1 bit /bit and expected of BBS.
If the answer is significantly less than 1 bit /bit, it's not random.