Key exchange resilience to DoS attack

Question

I am designing a user registration protocol for a system where I control both clients and servers. The aim is to establish a symmetric key for subsequent authentication, while minimizing the work load on the server side. The security concerns are less about privacy than about integrity of the system and resistance to DoS attacks. I thought about doing something like this:

1. Client generates an RSA public-private key pair
2. Client sends public key to server
3. Server returns randomly generated key, encrypted in the public key

Is there any risk that malicious clients may choose public keys that cause undue work load on the server? Or, other problems? Is there a better way to minimize load on the server in the face of malicious clients, while still insuring the security of the key exchange for cooperative clients?

Answer 1

Steps 5 and 7 remain very important even when one ignores DoS attacks.

4. $\;\;\;$ The RSA key-pairs use ​ e = 3
   
5. $\;\;\;$ Server signs $\: \langle \hspace{-0.03 in}$ RSA_modulus , RSA_ciphertext $\rangle \:$ and returns that to Client in step 3
   
6. $\;\;\;$ If computation is significantly more of a bottleneck than communication,
   $\;\;\;$ then Client computes a sufficiently accurate estimate of its modulus's
   $\;\;\;$ reciprocal in step 1 and sends that estimate to server in step 2
   
7. $\;\;\;$ In general, if step 6 is used then its estimate may need to go into what gets signed
   $\;\;\;$ in step 5. $\:$ However, I believe the most straight-forward way to use estimates from
   $\;\;\;$ step 6 would have the property that if an adversary changed it to a bad "estimate",
   $\;\;\;$ then Server will either detect that or get the right result anyway.
   $\;\;\;$ If step 6 is used and the use of its estimates has that property and the key(s)
   $\;\;\;$ is(are) only used as input to a secure MAC scheme and/or a stateless IND-CPA
   $\;\;\;$ encryption scheme and those schemes are only used for authentication
   $\;\;\;$ and/or (possibly-already-authenticated-)encryption respectively, then the
   $\;\;\;$ estimate from step 6 does not need to go into what gets signed in step 5.
   $\;\;\;$ (It's fine if the encryption is used in a way that keeps state, but I only see a proof for the
   $\;\;\;$ cases in which the underlying encryption scheme is IND-CPA despite not keeping state.)

Explanation of what's going on in #7:
By changing the honest client's estimate to a bad one, the adversary might be able to make
the event "server aborts" depend in a useful(-to-the-adversary) way upon what the resulting
key would otherwise have been and the randomness that would've been used to encrypt it.
The assumptions I gave are enough to guarantee that subsequent
communication remains secure despite such any such dependency.