(significantly reworded this question to fit better into crypto.se).
I have static RSA keypairs for both sides and an encrypted transport that needs symmetric keys, and need to exchange session keys. The library I base this on is OpenSSL, which doesn't seem to have some pre-canned AKC protocol for me to use, so I have to roll my own.
I thought about various options, all of which use RSA for authentication and session key exchange and an additional curve25519 ecdh exchange for perfect forward secrecy.
I wonder how complicated it needs to be. Right now I do this - to start a half-duplex connection (session and hmac keys for sending only), I send a fully random RSA challenge and the public ecdh key.
The other side responds by checksumming the decrypted rsa challenge and the ecdh key, thereby proving that it has the private key, and then enganges in exactly the same protocol in reverse, so there are two rsa and ecdh exchanges (this is because it is simpler than doing only one ecdh exchange, even if it is slower).
The devil is in the details, here are my questions:
- is it safe to use the random rsa challenge for key material (I use hkdf to derice keys)? I.e. is it safe to use the random rsa challenge data both for authentication and for encryption keys? My gues sis yes, but I would like to have an informed opinion, not mine :)
- how should the authentication proof (digest (rsa + ecdh)) be calculated? Is a straight sha512 safe? Right now, I use hmac(part-of-rsa-challenge, rsa-challenge + ecdh), but this might be overkill, and even detrimental, as the hmac key is part of the hashed challenge data.
- should the authentication proof include the ecdh data? If it does, then tampering could be detected earlier, if it does not, then the decryption fails later. My guess is that it doesn't really matter for the security.
- the rsa challenge includes the hkdf salt, and additional keying material used by both sides, and there would even be space to include the ecdh key in the rsa. is it best to take as little as possible from the rsa challenge and send e.g. ecdh and hkdf salt in cleartext, or is it best to have as much data inside the rsa challenge?
- Should I use a kdf to derive as many keys as possible from the same key material, or should I use as much independent key material as possible, e.g., when I have 1024 bit key material, should I use a single hkdf to derive 4 session keys from it, or should I use four independent hkdf invocations of each 256 bit part of the key material?
Most of these more or less boil down to the first question: how much (if any) of the random rsa challenge should be used for session key material.