Is there a way to break this encryption?

I was wondering about making up a simple system to encrypt a file:

seed the random number generator with the passcode
generate as many random bits as the file has
xor


This has a vulnerability that if you encrypt two file A and B with the same passcode, xoring the two encrypted files would give the xor of A and B... so

seed the random number generator with the passcode concatenated with a hash of the file
generate as many random bits as the file has
xor


I think this fixes that vulnerability but a MITM could still flip certain bits inside the file by flipping bits on the encrypted file, so

seed the random number generator with the passcode concatenated with a hash of the file
randomly permute the plaintext in blocks of 128 bits
generate as many random bits as the file has
xor


I couldn't think of any vulnerability this system has. Do you know of any?

-
what are you randomly permuting in the 3rd proposal? The plaintext? If so, how do you undo the permutation? Why not use a MAC to prevent an adversary from flipping bits? – mikeazo Jul 24 '12 at 18:58
@mikeazo, good point I would have to have the passcode and hash of the file to decrypt it! I overlooked that. – user2558 Jul 24 '12 at 19:00
@xce Under option 2 there is no way to decrypt the file since the decrypter doesn't have access to a hash of the file it can't generate the same random sequence and decrypt the file. – Ethan Heilman Jul 24 '12 at 21:50

This scheme isn't probabilistc so it could not be even CPA secure. Another weakness in this scheme is in the seed for PRG. the value "passcode"||"hashedvalue" is not uniformely distributed value. This mean that a CPA-attacker knows a part of the seed and then you cannot rely on the PRG property. A way to break the second weakness is to concatenate message and passcode and THEN hash them togheter. This could be at least a deterministic encryption scheme in the Random Oracle Model.

-

Your permutation idea is leaving the realm of encryption and trying to address a different type of idea: authentication. Encryption hides the bits, authentication gives confidence that they are being read as intended.

But it doesn't do this properly, the attacker might still be able to flip bits. What if the random permutation maps the modified ciphertext back to something that's valid? What if the attacker grabs a different ciphertext block that he knows the plaintext for and just plain overwrites a ciphertext block with it? Etc. Instead, if you want to be sure bits were not flipped, use the proper way to do so: a MAC.

At best your permutation idea will provide corruption in the decrypted plaintext similar to like what we see in CBC if a block is changed: Random corruption of one block. While this can be a somewhat pragmatic method of detecting tampering, it is by no means sufficient for the purposes of authentication.

-

For the 3rd proposal, see my comment to the question for the issues/questions I have regarding there.

For the 2nd proposal, basically what you are proposing is using an IV with a stream cipher. The IV in this case being the hash of the file which would have to be shared in order to decrypt. In theory, it is a good idea. In practice, however, the track record is not very good. WEP used an IV with RC4, which led to a serious vulnerability. Furthermore, of the 34 stream ciphers submitted to eSTREAM, 10 were broken due to the way they used the IV during setup.

This shows that simply putting an IV and a password together is not necessarily better or even secure. You would need much stronger guarantees (for an example of a security proof for IVs in stream ciphers, see On the Security of IV Dependent Stream Ciphers).

So, to answer your question, in general, the construction is not secure. For specific stream ciphers, it might be possible to prove security.

One practical issue I see with this specific construction (even if a proper stream cipher is used), is that if I encrypt the same file with the same key twice, the ciphertext is the same. This leaks some information to an attacker, which is never a good thing.

-
Using the hash of the file as a public IV is extremely dangerous since it allows an attacker to try plaintexts and detect if they match. XCE would need to add randomness to plaintext to avoid this, but why not just use a random IV instead. – Ethan Heilman Jul 24 '12 at 21:55