# Does decryption with a wrong key creates a new encryption?

I'm wondering if decrypting with a wrong key is the same a encrypting in a different way.

For example if I encrypt the message "hello" with Caesar cipher with key = 3 it will give the message "khoor" but if I decrypt it with key = 2 the result will be "ifmmp". So we can say, decrypting a message encrypted with Cesar cypher will be the same as encrypting the encrypted message with -wrongKey.

But in a general way, if I decrypt a message with a wrong key, do I encrypt the encrypted message in a different way?

• In general no. Decryption with a different key can fail, result in some output that is a valid ciphertext or result in an output that is not in fact a valid ciphertext. It depends on the encryption scheme and its security guarantees. – Maeher Dec 1 '20 at 13:13
• Do you have in mind an algorithm that can fail? And how does the decrypter can know if the result is good or no if it doesn't know the key or the good result? – Arthur Monteiro Dec 1 '20 at 13:26
• Any AEAD will return a failure code if the incorrect key is used, and not release any decryption results. AEADs are the recommended way to do almost all encryption. – SAI Peregrinus Dec 1 '20 at 15:34
• Take any authenticated encryption scheme and sample two keys independently and uniformly at random. Choose a random message and encrypt it under the first key. INT-CTXT security guarantees that decryption with the second key will fail with overwhelming probability. – Maeher Dec 1 '20 at 15:34
• @SAIPeregrinus as was just pointed out to me by SqueamishOssifrage in chat, that is not quite correct. For many AEAD it's perfectly possible to find a ciphertext and pairs of keys, such that the ciphertext is valid under both. However, it can of course still fail and will if the keys and encrypted message are chosen independently. – Maeher Dec 1 '20 at 15:37

In general this is a moot point, as modern encryption schemes are designed so that the recipient knows if they have used the correct key or not. It is an interesting question to ask about encryption primitives, like a block cipher. Put more precisely, you are asking whether there is always a $$k_3$$ such that

$$D_{k_2}(E_{k_1}) = E_{k_3}$$

This is essentially asking if encryption forms a group, and in general the answer is no. See e.g. Is Triple-DES a group?. There are notable exceptions. You've seen that Caesar cipher is one, and Vigenere is another.

• The paper linked is very interesting: link.springer.com/chapter/10.1007/3-540-48071-4_36 – cisnjxqu Dec 2 '20 at 10:42
• Do you know of any modern symmetric ciphers that form a group? – cisnjxqu Dec 2 '20 at 10:43
• How many iterations would be required to form a group? i.e. if I not only include $\mathrm{Enc}_{k_1}$ in my set of permutations but also $\mathrm{Enc}_{k_2} \circ \mathrm{Enc}_{k_1}$, or in general $\mathrm{Enc}_{k_n} \circ \dots \circ \mathrm{Enc}_{k_2} \circ \mathrm{Enc}_{k_1}$ - what is the smallest $n$ such that the set of permutations forms a group? – cisnjxqu Dec 2 '20 at 10:46
• @cisnjxqu Why is double encryption that's equivalent to single encryption no better than single encryption? also, the Pohlig-Hellman cipher has the double encryption property. – kelalaka Dec 2 '20 at 19:48
• @cisnjxqu I believe there's no reason to think that any modern cipher forms a group, but it's hard to prove either way. – bmm6o Dec 3 '20 at 17:47

Generally, as mentioned in the comments, if you use an authenticated cipher with an independently chosen key, then decryption will fail with very high likelihood, and can therefore not be considered encryption. It's the same thing basically with padded modes such as CBC. If you'd use it with padding then the decryption will fail with high probability (although once in ~256 is not as high as you would expect with an authenticated cipher).

On the other hand, with CTR mode and most stream ciphers, encryption = decryption by definition (XOR of a key stream with the plaintext / ciphertext). And for block ciphers (which, by themselves, are not semantically secure), you can use them in both directions. In that case yes, your premise is correct. Note that this answer disregards how the nonce or IV is handled (it can e.g. be established out of band).

I'll not go into asymmetric ciphers much; lets us please agree that encryption with a private key is usually not possible.

So it entirely depends on the cipher scheme.

• Well, with CTR mode the ciphertext should include a nonce. Depending on how exactly this is encoded, this nonce is either missing after decryption, or your ciphertext has become shorter. – Maeher Dec 1 '20 at 23:21
• Comments are not for extended discussion; this conversation has been moved to chat. – Maarten Bodewes Dec 2 '20 at 8:32