What are the major flaws and limitations of this hash-based stream cipher design?

Question

• K is a 224-bit key.
• C is a 32-bit counter; it is publicly known. the same value is never used twice.
• plaintext has arbitrary length (up to 64kB).
• H is a cryptographic hash function that returns len(plaintext) bits.
• ^ is the bitwise XOR.
• + is the concatenation.

Encryption:

ciphertext = plaintext ^ H(K + C)

Decryption:

plaintext = ciphertext ^ H(K + C)

• Are there any fundamental flaws?
• Would it make sense to use a longer key?
• Under what circumstances would doing H(C + K + C) instead of H(K + C) make the algorithm more secure?
• I intend to use Skein, which comes in three flavors of internal state sizes: 256, 512 and 1024 bits. How does the internal state size limit the strength of the whole system? Would it make any sense to use the 512/1024 bit variants, even though len(K+C) = 256?
• Is there anything particular about Skein that would make it unsuitable for the job?

edit: Thinking about it, len(K) > internal state size seems pointless. To break the encryption, an attacker would just need to find a K that produces the same internal state as the real K.

Answer 1

Are there any fundamental flaws?

The construction itself is secure for a good H. It's close to what many stream ciphers use internally. E.g. chacha has an H that hashes a 256-bit key, a 64-bit nonce and a 64-bit counter to get 512 bits of output.

Would it make sense to use a longer key?

Yes, if you want more brute force resistance, since finding a preimage is not enough to find the key, if it leads to different values for other counter values.

Under what circumstances would doing H(C + K + C) instead of H(K + C) make the algorithm more secure?

If your H was vulnerable to length extension attack (e.g. SHA-256) and the counter was variable length, H(K + C) would not be secure, but H(C + K + C) might. However, with a constant sized C, I can't think of a reason to use H(C + K + C).

I intend to use Skein, [...]

Then shouldn't you use the stream cipher construction from the Skein paper (pdf, section 4.10)?

Answer 2

…limitations of this hash-based stream cipher design?

One of the limitations of this hash-based stream cipher design would be that it does not offer any kind of “authentication”, nor any kind of “integrity-protection”.

Maybe you just forgot to describe how you will be handling authentication and integrity-protection, but it’s important not to forget about it.

I intend to use Skein…

That raises the question why you aren’t simply using Threefish as a well-vetted block cipher. After all, the design of Skein is based on the Threefish tweakable block cipher!

In the unlikely case you really need a stream-cipher because something prevents you from using a block-cipher, you could create one based on Threefish, which is bound to provide a bit more speed (when implemented correctly) than building a cipher based upon the Skein Hash Function Family.

Nota Bene:

As it’s not clear to me why someone would try to create a hash-based stream cipher while well-analyzed stream ciphers are available, I have skipped cryptanalysis – which means that your design may (or may not) have security issues I did not check on. In the end, I guess I somewhat chime in with Stephens comment and simply don’t seem to have that “scientific interest” you mentioned in your comment.