I have a couple of related questions:
I read about an estimate, ascribed to Shannon, of the entropy of English text to be between 0.6 and 1.3 bits per letter. The modern day corpus, including the huge and exponentially ever expanding volume of materials accessible via the Internet, is different from the one available a century back. Is there a revised estimate?
If one simply takes two books from one's small private library (that contains e.g. only very common novels) and arbitrarily select two starting points to combine with xor the letters, then the resulting pseudo-random sequence presumably could be argued to have a non-negligible risk of being attacked through brute-force by a knowledgeable opponent. But the actual choice of materials (including especially those via the Internet) to be combined is nowadays virtually unlimited. If one could thus appropriately choose a sufficiently large number of text sources to be combined, wouldn't the result be practically secure for crypto usages in view of the combinatorial explosion of the complexity that the attacker has to deal with? One could certainly also employ some more complex operations than xor and even a couple of different operations either simultaneously or in variations. Are some good schemes in this direction already being used in non-trivial practical applications?
[Added on edit:] A later work of Cover and King give an estimate of 1.34 bit per letter, which implies that appropriately combining 4 text files should lead to sequences of full entropy. Several weeks ago I wrote a software for that but unfortunately, due to a bug in codes used in its design, it had to be retracted for not being satisfactory. I have just released a carefully tested replacement, named TEXTCOMBINE-REV, in which I wrote in Prologue:
What has been achieved by the present software can be tersely summarized as follows, assuming the general case where the text files are sufficiently large:
(1) The generated byte sequences pass, via design specifications of the software, Maurer's universal test and the autocorrelation test for all d in the range [1, 16] as well as the ENT test with an entropy value according to it of at least 7.99 bits per byte. The software is namely coded such that it would give up, reporting failure, after a certain specified maximum amount of processing has been done without finding a solution.
(2) An extensive expermiment of the present author done on all different combinations, totalling 3060 in number, of 4 source materials (of size 600 KB each) taken from 18 different books of English literature downloaded from Project Gutenberg resulted in the following:
(a) No case of failure was ever encountered. On the contrary, the above mentioned processing limit, which is in terms of rounds of certain preprocessing of source materials before they are xor-ed together, was by far not being approached in the experiment. For details, see Epilogue.
(b) The worst case of entropy according to ENT in the experiment was higher than 7.995 bits per byte and the average CPU-time was less than 15 sec on author's PC.
The software (new version 1.1) is available at http://mok-kong-shen.de