Is it secure to use RSA to exchange AES key?

Question

I have to create a Client-Server Application using Java and I want to make the communication secure. I thought to use AES to encrypt the messages and for key exchange I make the next steps:

• Client generates RSA keys and send public key to server
• Server will encrypt the AES key with RSA Public key (AES key is generated random for every client)
• Client decrypt the message with its RSA Private key and then all messages will be encrypted with AES

My question is: is it a good practice ?

Answer 1

No, it is not a good practice.

1. It is prone to man in the middle attack. When server receives a public key, it does not know if this key comes from a client or from the "man in the middle". Thus, the "man in the middle" can intercept the traffic between client and server, for client it can simulate server, for server it can simulate client. Thus, the whole communication will be 1) known to MitM and 2) data sent in both directions can be modified by MitM.

• --> You can avoid it, if you authenticate clients to server or authenticate server to clients by using public key certificates.

2. It is prone to replay attack. Suppose a client sends command "transfer 1000 USD to account 123456789". This can be legitimate operation. But if an attacker intercepts the traffic and resends it within relatively short time (so that the server still uses the same key for AES encryption for this client), then server will not be able to distinguish, if this traffic comes from the client of from the attacker, and will execute the command. Even if you authenticate clients to server or authenticate server to clients, this problem will still persist.

• --> You can avoid it, if you use nonce.

3. The good thing: Your current approach has relatively good forward secrecy: past sessions are protected against future compromises of client keys or session passwords. But if you address the problem 1 mentioned above (authentication) and start using the same RSA key over multiple sessions, then you will lose forward secrecy. If an attacker obtains private key, all past sessions can be decrypted. To prevent it, you will need further measures, e.g. ephemeral Diffie-Hellman key exchange.

4. Independent on all the problems above, it is important to properly use RSA, as @kelalaka mentioned. See details here and here.

Thus, it may be much more secure to use TLS implementation provided by Java, instead of implementing your own protocol.