I am designing a user registration protocol for a system where I control both clients and servers. The aim is to establish a symmetric key for subsequent authentication, while minimizing the work load on the server side. The security concerns are less about privacy than about integrity of the system and resistance to DoS attacks. I thought about doing something like this:
- Client generates an RSA public-private key pair
- Client sends public key to server
- Server returns randomly generated key, encrypted in the public key
Is there any risk that malicious clients may choose public keys that cause undue work load on the server? Or, other problems? Is there a better way to minimize load on the server in the face of malicious clients, while still insuring the security of the key exchange for cooperative clients?