I have a potentially infinite stream of data originating in some physical phenomena that may expose random behavior (due to quantum effects, thermal noise, the butterfly effect in chaotic dynamical systems, etc). The stream may carry some amount of entropy (true randomness), but it may be distributed unevenly throughout the stream and its density is unknown.
I have a limited storage space (say, 1GB). I need a procedure that can collect as much entropy as possible from the stream to the storage provided, and, when required, generate (in a reasonable time) a single file containing the collected entropy. It is preferable to make the resulting file as small as possible without reducing the amount of entropy it contains.
How can I implement such a procedure?