I have a client and a server communicating with messages encrypted using RSA.

Initially, the client knows server's public key, but the server does not know the client's public key. Is it possible to securely receive the client's public key over a network (i.e. without a man in the middle intercepting and changing the public key the client is trying to send with their own).

I was thinking that the client can generate a random secret and then send it to the server using the server's public key for encryption. Then the server can decrypt the secret, use it as a key for AES, and echo it back (this time encrypted with AES). The client then verifies that the decrypted secret is the same as the one it sent.

I'm scratching my head trying to figure out if this is secure, and how I should incorporate hashing to verify the integrity of the data.. Would this even work?

  • $\begingroup$ I think it is possible to send it securely as you have the server's public key. Just negotiate a session key and send it securely (it's what HTTPS does all the time). The problem is not sending it securely, but how does the server know that the client is who he is supposed to be. $\endgroup$
    – xxxxxxxxx
    Feb 5, 2014 at 7:23
  • $\begingroup$ I guess my question is how do I negotiate the session key? That is the big part I am missing. I only want to use RSA (and AES if I need to). $\endgroup$ Feb 5, 2014 at 7:30
  • $\begingroup$ Does the client initially have any secret information that can be used to prove it is genuine? If not, you obviously can't prevent a man in the middle intercepting and changing the public key the client is trying to send with their own, at least using cryptography (including public-key and quantum). $\endgroup$
    – fgrieu
    Feb 5, 2014 at 7:39
  • $\begingroup$ More accurately, I have multiple clients and a server. Who each client is specifically isn't important information for me. The only important thing for me is that I can establish a secure connection between the server and a new client. I'm not really trying to prevent a man in the middle changing the public key. I know there's nothing I can do to stop that. I want to be able to detect when it happens and then halt communications. $\endgroup$ Feb 5, 2014 at 7:56
  • $\begingroup$ This is why I suggested I generate a secret on the client that only the server can decrypt (using its private key) and then echoing the secret back to the client using symmetric encryption (with the secret as the key). If a MITM replaces the secret with their own, they still won't know what the client's secret was (only the server can know that) so verification will fail back on the client end when it receives the wrong key back from the MITM (or the server). I think I need to add hashes in here somewhere but I haven't really fleshed the idea out yet, and I don't even know if this is viable. $\endgroup$ Feb 5, 2014 at 8:03

2 Answers 2


In this protocol:

  1. client draws a random AES key $K_\text{SES}$ (I'll make it 128-bit);
  2. client enciphers $K_\text{SES}$ using the server's RSA publickey $K_\text{PUB}$, and sends the result $\text{RSAES}_{K_\text{PUB}}(K_\text{SES})$ (note: RSAES designates RSA encryption with some unspecified padding e.g. per one of the two fine encryption schemes defined in PKCS#1);
  3. server receives the alleged $\text{RSAES}_{K_\text{PUB}}(K_\text{SES})$ and deciphers it into the alleged $K_\text{SES}$, using the server's RSA private key $K_\text{PRI}$; servers stops if that fail;
  4. server enciphers $K_\text{SES}$ using AES key $K_\text{SES}$, and sends the result $\text{AESENC}_{K_\text{SES}}(K_\text{SES})$;
  5. client receives the alleged $X=\text{AESENC}_{K_\text{SES}}(K_\text{SES})$, deciphers it using the original AES key $K_\text{SES}$, and aborts if $\text{AESDEC}_{K_\text{SES}}(X)\ne K_\text{SES}$;
  6. presumably, $K_\text{SES}$ is then used as a session key; I'll assume as key to some unspecified mode of AES for confidentiality, and perhaps something (not even suggested) for authentication; at least the client's public key is sent using that encrypted channel.

A 5-minute non-exhaustive analysis reveals various serious issues, limitations, and risks with this protocol:

  • there is no explicit protection against replay of a session to the server; in many circumstances this is a gapping hole!! (clarification: replay to the server is simply sending the messages sent by a genuine client and recorded in an earlier session, including any IV, and would work in the absence of some challenge-response from the server, which is currently not described);
  • it was imprecisely specified (it still is, in particular regarding error handling and time-outs), making a detailed analysis impossible;
  • it is not peer-reviewed, thus must be presumed unsafe;
  • as any protocol starting without secret credentials on the client side, it can not authenticate the client;
  • as any known non-quantum protocol, it can not prevent Eve-in-the-middle from intercepting or/and relaying the messages, but that is not by itself a security issue;
  • there is no protection against denial of service of the server by multiple connections, replayed or not;
  • there are Bleichenbacher padding oracle attacks on some bad implementations of step 3. on the server side (typically when using RSAES-PKCS1-V1_5), that in the worst case could leak $K_\text{SES}$, and then the sky is the limit;
  • if somewhat Eve-in-the-middle manage to abuse the implementation of step 6. on either the client or server side as a decryption oracle, then the possibility exists that she can obtain the decryption of $X=\text{AESENC}_{K_\text{SES}}(K_\text{SES})$ using the active key $K_\text{SES}$, giving her $K_\text{SES}$, with similarly devastating effect (clarification: if the encryption uses CBC, the attack could consist is replacing one genuine ciphertext block with with $X$ obtained from step 4. and somehow obtain the corresponding plaintext block, that would yield $K_\text{SES}$ after an exclusive-or with the previous ciphertext block ; that should not be possible in an authenticated link, but not enough details about that are given);
  • if $K_\text{PRI}$ ever leaks, the protocol is entirely insecure after the leak with no stated provision to recover from that;
  • if $K_\text{PRI}$ ever leaks, the confidentiality of past recorded communications is compromised; contrast that with protocols offering forward secrecy;
  • update: there is no stated safeguard to reduce the risks resulting from a poorly seeded or otherwise bad RNG being used at step 1., which times and times has occurred on the field;
  • update: at step 6. the client's public key is sent enciphered, when we really need authentication, not really described.
  • $\begingroup$ Thanks for the feedback! I am using RSA padding, and I would be using AES with CBC and a random IV so replay attacks shouldn't work (unless I misinterpreted what "replaying a session" is. I am still very very new to cryptography). About DoS.. Maybe I can have a proof of work problem the client must solve first before I am willing to establish communications.. Good tip, didn't even think about that. I will be using PKCS1 2.1 for padding so the padding attack isn't a problem (at least not the one for version 1.5). $\endgroup$ Feb 5, 2014 at 15:45
  • $\begingroup$ As for the problem with step 6... I don't really understand this, I will have to look up what a decryption oracle is. Also, I may just ditch AES and use RSA for the client as well (send its public key instead of a secret key used for AES). Would this be a better option? $\endgroup$ Feb 5, 2014 at 15:47
  • $\begingroup$ @user3100783: I addressed your comments in a revision of the answer, hopefully clarifying replay, and how step 6. could be abused. Using RSA for bulk encryption (that is, of something that does not fit a block) is very seldom (if ever) a good idea. Also, you want to send the client's public key, therefore there is no reason to encipher it in the first place. $\endgroup$
    – fgrieu
    Feb 5, 2014 at 17:02
  • $\begingroup$ Thanks for the help, I think I know how to address all these problems except for authentication. There is no secret that the client has that the server can know about before initial connection. The client will be unknown to the server until its first connection. I guess that after the first connection, I can send something unique to the client to the server so on later connections this can be used for later authentication (now that the server knows the client). Before that, it's not really possible since there is nothing to authenticate. $\endgroup$ Feb 6, 2014 at 3:37
  • 1
    $\begingroup$ The protocol is vulnerable to an attack where the attacker can make both the server and the client believe they have exchanged a correct session key while the attacker has exchanged another one with the server and can send him correct messages if he knows the protocol. You need to add identifiers and any kind of authentication to your messages to prevent this attack. $\endgroup$
    – Ekris
    Feb 6, 2014 at 22:53

If clients authentication is no necessary, then this protocol looks safe. But I have got couple suggestions. Use RSA-OAEP padding scheme, for making communication between client and server more secure. This scheme could prevent MITM changing message without decryption.
Another way to make your protocol more secure is adding to server response digital signature, to confirm that received message exactly from server.

  • 3
    $\begingroup$ I seriously suggest that anyone refrain from giving positive opinions on any protocol before both: years of study of why past similar recommendations turned out dead wrong; and days of study of the proposed protocol based on a detailed description (I occasionally succumb to shortening that later period, but only when not paid for the job). $\endgroup$
    – fgrieu
    Feb 5, 2014 at 17:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.