Public key encryption and message integrity

Question

Let's say I want to send an encrypted email thus I encrypt it with the receiver's public key, so that the receiver uses its private key to decrypt it.

If someone tampers with the message in between then

1. what kind of tampering can be performed?
2. wouldn't the decryption process fail since the message ought to be malformed due to the tampering?

Does asymmetric encryption like RSA ensure message integrity without hashing the message with an HMAC? Or is hashing necessary only when utilizing symmetric encryption?

For example at initiation time of the TLS handshake when the client encrypts with the server's public key the symmetric key that's going to be used for the rest of the transaction, does it also uses hashing?

Answer 1

let's say I want to send an encrypted email thus I sign it with the receiver's public key

Huh? With the standard definition of sign, you sign with a private key; it doesn't make sense to sign with a public key, in part because anyone with access to that public key can do it.

I think you mean 'I encrypt it with the receiver's public key'

If someone tampers with the message in between a. what kind of tempering can be performed

One of the things they can do is pick a message, and encrypt it with the receiver's public key (which, presumably, they have - it's public), and then replace your ciphertext with theirs. Then, when the receiver gets it, it looks just fine, without the message being malformed (because the message was encrypted properly).

Does asymmetric encryption like RSA ensure message integrity without hashing the message with an HMAC?

Any form of public key encryption, in and of itself, cannot provide any guarantee of message integrity; if you need such a guarantee, you need to provide something in addition.

For example at initiation time of the TLS handshake when the client encrypts with the server's public key the symmetric key that's going to be used for the rest of the transaction, does it also uses hashing?

Actually, it doesn't (TLS does do hashing as a part of the KDF, but that's not what you're asking about).

Assuming that TLS doesn't do client authentication (which is optional), the server doesn't get any sort of integrity check; when client Alice selects a premaster secret, encrypts it and sends it to server Bob, Mallet could indeed replace the encrypted premaster secret with his own, and Bob won't know (or case); as far as Bob is concerned, he just set up a secure connection with Mallet. However, Alice won't establish a secure connection with anyone (Bob never heard her encrypted premaster secret, and Mallet can't decrypt it), and so as far as Alice is concerned, the TLS connection failed (which is the secure option in this case).

Answer 2

If someone tampers with the message in between a. what kind of tampering can be performed and b. wouldn't the decryption process fail since the message ought to be malformed due to the tampering?

That depends on the format. In general, yes, tampering is detected. That may even be the reason that the protocol is insecure, as it could lead to padding oracle attacks.

Note that an attacker can also simply encrypt a different message as encryption doesn't add integrity or authenticity, and the public key is commonly considered, well, public.

c. Does asymmetric encryption like RSA ensure message integrity without hashing the message with an HMAC? or is hashing necessary only when utilizing symmetric encryption?

No. Generally the message is first signed and then encrypted. If a symmetric key is already shared the parties might as well encrypt with it - or with a key derived from it.

If a symmetric key is wrapped (encrypted) and sent, you need to verify that the secret key has been agreed upon successfully - as explained in the next section.

For example at initiation time of the TLS handshake when the client encrypts with the server's public key the symmetric key that's going to be used for the rest of the transaction, does it also uses hashing?

No, but as the symmetric key is later used for a HMAC over the previously communicated messages, it doesn't matter if the symmetric key was changed. If it was, the authentication tag calculated using HMAC value will be incorrect and the handshake will fail.

Answer 3

If you use the the right kind of public-key encryption then tampering should not be a problem. Note that for public-key encryption the adversary could always use the public key to encrypt any message he wishes. I.e., tampering with a ciphertext is mainly a problem if it can produce a new ciphertext that encrypts plaintext that is somehow related to the original plaintext.

Therefore, in this case you should use a public-key encryption scheme which meets the security definition of non-mallebility under apdaptive chosen ciphertext attacks (NM-CCA2). Essentially, this definition exactly means that an adversary will not be able to tamper with a ciphertext to produce a ciphertext that encrypts a plaintext that is in any way related to the original plaintext.

It turns out that NM-CCA2 security is equivalent to the better known security definition of indistinguishability under adaptive chosen ciphertext attacks (IND-CCA2). An example of such scheme is RSA with OAEP padding.

For details on these definitions and relations you can see this paper