DIY over-encryption with a related key

Question

I know next to nothing about cryptography, which is bad because I'm a programmer. I was exploring some code the other day and the encryption part of it seemed a little strange to me. I'd like the name/terminology/links to this algorithm/concept so I can learn more about it, and I'd also like to know if it's secure or a waste of processing. Also please bear with me, I don't know any of your terminology u.u

The code included some stuff that sounds pretty good: it uses CBC with AES/Rijndael (using 256 bit block size) with null byte padding (the data is supposed to end with a certain character so null bytes works). The IV is randomly generated with each encryption and is then prepended to the encrypted data (second question: does this practice have a name?). But then as a wrapper around that it has another layer of "encryption"? - I'll try to explain it the best I can:

First, the key that was used in the AES/Rijndael is passed through a hash function. I'll call this the key hash? This same value is used in both encrypting and decrypting. Then the encrypted data (from the AES CBC, and with the IV in front) is broken into chunks the same length of the key hash (blocks?). Each byte of the key hash is added to the encrypted byte, and then modulus-ed(?) (if > 255 then subtract 255). Currently the hash function used is MD5, but I'd also like to know about this in a more general sense too.

Is this whole method secure? What is it called? Is it a waste of processing? Can it be done more efficiently/better? Why did the developer choose to do it like this? Etc... So many questions!

Not important but: The developer then proceeds to store/send the encrpyted data over the network as... doubly base64 encoded data, and then inside xml. e.g. xml(base64(base64(encrypted data)))... Gross!!! And the original data is so verbose and redundant too.

Answer 1

The key that was used in the AES/Rijndael for encryption is passed through a hash function

^{(strikethrough/strikeout mine)}

That's OK if the hash function is secure and not vulnerable to timing attacks. Obviously MD5 is not a great option as it has been proven to be insecure. For this specific function however - similar to key derivation - it's OK. You should not be able to retrieve the key value as a hash is a one way function.

I'll call this the key hash? This same value is used in both encrypting and decrypting. Then the encrypted data (from the AES CBC, and with the IV in front) is broken into chunks the same length of the key hash (blocks?). Each byte of the key hash is added to the encrypted byte, and then modulus-ed(?) (if > 255 then subtract 255).

This is basically a one-time pad where the key is repeated. Now normally it would be relatively easy to find the "hash" key with a known plaintext attack. With the ciphertext being properly randomized that is not possible. If however some part of the original AES/CBC ciphertext is ever leaked then the hashed key becomes known and the cipher can be reversed.

Basically this scheme doesn't add much security. It also shows a very limited understanding of cryptographic techniques and encoding methods. Because of this it should be seen as a risk, even if the scheme cannot be attacked directly. If MD5 is ever reversed you're in trouble. For now it is indeed a waste of bandwidth and processing time.

Furthermore, if you'd ever want to use hardware based encryption then the encryption key value will not be directly available (as the encryption operation is performed in the hardware itself). So it's probably not possible to perform MD5 over the key, making this scheme incompatible with hardware based security.

Finally, there seems a lot to be going on, but this scheme in all its needless complexity seems to miss an authentication tag. This means that an attacker may send you corrupted data that's still successfully decrypted. The attacker won't be able to simply inject any data, but it is possible to at least let the receiving party process garbage. In that sense the hash function could have been put to better use in a HMAC construction. In that case a SHA-2 hash such as SHA-256 should be preferred of course.

Answer 2

First of all,

I know next to nothing about cryptography, which is bad because I'm a programmer.

No this isn't bad as you say. I know many programmers that know nothing about crypto but a lot in other fields (data informations, parallel computing, machine learning...) So nothing wrong here.

---

It uses CBC with AES/Rijndael (using 256 bit block size) with null byte padding.

AES/Rijndael block size is 128 bit. The key size however can be 128, 192 or 256 bits. CBC means Cipher Block Chaining. The idea is to provide a sort of randomization of the data before they get encrypted. The padding is used to fill the last block.

The IV is randomly generated with each encryption and is then prepended to the encrypted data (second question: does this practice have a name?).

The use of the IV is to provide a randomization on the resulting cipher text. The idea is that if you encrypt the plain text $m$ more than one time : ($E(m,K)$ and some time later $E(m,K)$, not $E(E(m,K),K)$). The cipher text should not be the same. This prevent information retrieval on the first block.

---

Here is what happen during full encryption process.

Is this whole method secure? Is it a waste of processing? Can it be done more efficiently/better?

No, it is not more secure. It does not have a name. The layer of security provided by AES is sufficient, therefore it is a waste of processing.

Moreover, the use of MD5 here is clearly a joke. For a similar idea one should better use Keccak as a stream cipher : generate from the key a hash long enough to be Xor with the message.

PS : Rolling your own crypto is a bad idea.