# Public key encryption and message integrity

Let's say I want to send an encrypted email thus I encrypt it with the receiver's public key, so that the receiver uses its private key to decrypt it.

If someone tampers with the message in between then

1. what kind of tampering can be performed?
2. wouldn't the decryption process fail since the message ought to be malformed due to the tampering?

Does asymmetric encryption like RSA ensure message integrity without hashing the message with an HMAC? Or is hashing necessary only when utilizing symmetric encryption?

For example at initiation time of the TLS handshake when the client encrypts with the server's public key the symmetric key that's going to be used for the rest of the transaction, does it also uses hashing?

• TL;DR: If your asymmetric encryptions is decent (e.g. newer than like 10yrs ago), then a ciphertext modification will definitely be detected otherwise it depends on the details of what you're doing exactly. – SEJPM Sep 10 '16 at 19:15
• @sejpm what is the industry standard, like used in TLS? Is it open for negotiation between the browser and the server, and how can I ensure it uses a cyphersuite that is modern and tamper proof? – microwth Sep 10 '16 at 19:23
• The standard for asynchronous public key encryption (like for emails) it is S/MIME (using CMS as underlying primitive) or PGP. If bidirectional ("real-time") communication is required, TLS is preferred (using ECDHE+AES-GCM). – SEJPM Sep 10 '16 at 19:25
• @sejpm in the perfect forward secrecy is ONLY the Diffie Hellman key exchange + symmetric encryption used? I mean does Diffie Hellman+symmetric complement RSA or is it a case of "choosing one over the other"? – microwth Sep 10 '16 at 19:30
• If and only if (for now) a cipher suite contains the string DHE or ECDHE they provide forward secrecy. The "RSA" in this specifies the server authentication (e.g. the server certificate). (EC)DHE is then used to negotiate a key, RSA is used to make sure you're talking to the right server (using certificates and a signature) and symmetric crypto is used for the rest. – SEJPM Sep 10 '16 at 19:40

let's say I want to send an encrypted email thus I sign it with the receiver's public key

Huh? With the standard definition of sign, you sign with a private key; it doesn't make sense to sign with a public key, in part because anyone with access to that public key can do it.

I think you mean 'I encrypt it with the receiver's public key'

If someone tampers with the message in between a. what kind of tempering can be performed

One of the things they can do is pick a message, and encrypt it with the receiver's public key (which, presumably, they have - it's public), and then replace your ciphertext with theirs. Then, when the receiver gets it, it looks just fine, without the message being malformed (because the message was encrypted properly).

Does asymmetric encryption like RSA ensure message integrity without hashing the message with an HMAC?

Any form of public key encryption, in and of itself, cannot provide any guarantee of message integrity; if you need such a guarantee, you need to provide something in addition.

For example at initiation time of the TLS handshake when the client encrypts with the server's public key the symmetric key that's going to be used for the rest of the transaction, does it also uses hashing?

Actually, it doesn't (TLS does do hashing as a part of the KDF, but that's not what you're asking about).

Assuming that TLS doesn't do client authentication (which is optional), the server doesn't get any sort of integrity check; when client Alice selects a premaster secret, encrypts it and sends it to server Bob, Mallet could indeed replace the encrypted premaster secret with his own, and Bob won't know (or case); as far as Bob is concerned, he just set up a secure connection with Mallet. However, Alice won't establish a secure connection with anyone (Bob never heard her encrypted premaster secret, and Mallet can't decrypt it), and so as far as Alice is concerned, the TLS connection failed (which is the secure option in this case).

• >I think you mean 'I encrypt it with the receiver's public key' Yes that what I meant,I'll edit the question – microwth Sep 10 '16 at 19:06
• >then replace your ciphertext with theirs. However when not encrypting but signing i.e when Bob signs the message with his private key and Alice uses his public key to verify it, does ensure authentication but it's not tamper proof too? – microwth Sep 10 '16 at 19:07
• @microwth: we assume that no one other than Bob can generate a valid signature for a message; if someone changes the message in any way, the signature will not verify. If Alice verifies the signature, she knows that either a) Bob signed that exact message as promised, b) somehow Alice was tricked into using a public key that wasn't actually Bob's, or c) someone reused a signature (and message) that Bob signed previously. A valid protocol will make sure that (b) and (c) are not feasible alternatives. – poncho Sep 10 '16 at 19:21

If someone tampers with the message in between a. what kind of tampering can be performed and b. wouldn't the decryption process fail since the message ought to be malformed due to the tampering?

That depends on the format. In general, yes, tampering is detected. That may even be the reason that the protocol is insecure, as it could lead to padding oracle attacks.

Note that an attacker can also simply encrypt a different message as encryption doesn't add integrity or authenticity, and the public key is commonly considered, well, public.

c. Does asymmetric encryption like RSA ensure message integrity without hashing the message with an HMAC? or is hashing necessary only when utilizing symmetric encryption?

No. Generally the message is first signed and then encrypted. If a symmetric key is already shared the parties might as well encrypt with it - or with a key derived from it.

If a symmetric key is wrapped (encrypted) and sent, you need to verify that the secret key has been agreed upon successfully - as explained in the next section.

For example at initiation time of the TLS handshake when the client encrypts with the server's public key the symmetric key that's going to be used for the rest of the transaction, does it also uses hashing?

No, but as the symmetric key is later used for a HMAC over the previously communicated messages, it doesn't matter if the symmetric key was changed. If it was, the authentication tag calculated using HMAC value will be incorrect and the handshake will fail.

If you use the the right kind of public-key encryption then tampering should not be a problem. Note that for public-key encryption the adversary could always use the public key to encrypt any message he wishes. I.e., tampering with a ciphertext is mainly a problem if it can produce a new ciphertext that encrypts plaintext that is somehow related to the original plaintext.

Therefore, in this case you should use a public-key encryption scheme which meets the security definition of non-mallebility under apdaptive chosen ciphertext attacks (NM-CCA2). Essentially, this definition exactly means that an adversary will not be able to tamper with a ciphertext to produce a ciphertext that encrypts a plaintext that is in any way related to the original plaintext.

It turns out that NM-CCA2 security is equivalent to the better known security definition of indistinguishability under adaptive chosen ciphertext attacks (IND-CCA2). An example of such scheme is RSA with OAEP padding.

For details on these definitions and relations you can see this paper