I've already read this: Is it feasible to build a stream cipher from a cryptographic hash function?
However, my proposed construction differs…
Suppose the hash generates N bits. These bits are split into two parts:
- $S$ bits are kept secret,
- $X$ bits are used for the encryption, xor'ing the bytes the usual way
- generate a random key $K$ that will be shared between the peers
- generate $H_0 = hash(K)$
- encrypt bytes by xor'ing them w/ bits in $X$. the original bytes are buffered in $X'$.
- after $X/8$ bytes, calculate $H_1 = hash(S|X')$
- goto 2.
Even if the attacker has the plaintext and thus can retrieve the bits in $X$, say in the first round, since $S$ is unknown, he will not be able to calculate $H_1$. If $S$ is sufficiently large, guessing/brute forcing will be unfeasible. On the receiver side, if the stream was tampered, the bytes decoded in that round will have the same bit errors, but then in the next round, the hash's avalanche effect will kick in, making decoding bytes in the next rounds impossible.
The bytes used for xor'ing could come directly from the hash's state (keccak comes to mind), and the original bytes could go there, too, performing the specific bit mixing operations of the hash after each cipher round.
For SHA256, 128 bits could be used for $S$, and 128 bits for $X$. Or, 64 bits for $S$ and 192 bits for $X$. The later would result in less processing per byte, with somewhat less security.
What is your take on this?