(significantly reworded this question to fit better into crypto.se).

I have static RSA keypairs for both sides and an encrypted transport that needs symmetric keys, and need to exchange session keys. The library I base this on is OpenSSL, which doesn't seem to have some pre-canned AKC protocol for me to use, so I have to roll my own.

I thought about various options, all of which use RSA for authentication and session key exchange and an additional curve25519 ecdh exchange for perfect forward secrecy.

I wonder how complicated it needs to be. Right now I do this - to start a half-duplex connection (session and hmac keys for sending only), I send a fully random RSA challenge and the public ecdh key.

The other side responds by checksumming the decrypted rsa challenge and the ecdh key, thereby proving that it has the private key, and then enganges in exactly the same protocol in reverse, so there are two rsa and ecdh exchanges (this is because it is simpler than doing only one ecdh exchange, even if it is slower).

The devil is in the details, here are my questions:

  • is it safe to use the random rsa challenge for key material (I use hkdf to derice keys)? I.e. is it safe to use the random rsa challenge data both for authentication and for encryption keys? My gues sis yes, but I would like to have an informed opinion, not mine :)
  • how should the authentication proof (digest (rsa + ecdh)) be calculated? Is a straight sha512 safe? Right now, I use hmac(part-of-rsa-challenge, rsa-challenge + ecdh), but this might be overkill, and even detrimental, as the hmac key is part of the hashed challenge data.
  • should the authentication proof include the ecdh data? If it does, then tampering could be detected earlier, if it does not, then the decryption fails later. My guess is that it doesn't really matter for the security.
  • the rsa challenge includes the hkdf salt, and additional keying material used by both sides, and there would even be space to include the ecdh key in the rsa. is it best to take as little as possible from the rsa challenge and send e.g. ecdh and hkdf salt in cleartext, or is it best to have as much data inside the rsa challenge?
  • Should I use a kdf to derive as many keys as possible from the same key material, or should I use as much independent key material as possible, e.g., when I have 1024 bit key material, should I use a single hkdf to derive 4 session keys from it, or should I use four independent hkdf invocations of each 256 bit part of the key material?

Most of these more or less boil down to the first question: how much (if any) of the random rsa challenge should be used for session key material.

  • $\begingroup$ I only now realised that this qustion is rather practical, and thus should probably be asked on security.se, not crypto.se. I can reword the question to only discuss the planned protocol itself, which would probably fit crypto.se. $\endgroup$ Commented Jul 16, 2013 at 10:55
  • $\begingroup$ I reworded the question to not rely on openssl anymore. $\endgroup$ Commented Jul 17, 2013 at 10:33

1 Answer 1


The standard advice in this situation would: use a well-vetted mechanism, like TLS (or DTLS, or some other similar scheme). If you're not doing that, you'd better have a good reason, as designing your own key-exchange protocol is non-trivial and unnecessarily risky.

  • $\begingroup$ I agree (see the first part of my question), but neither TLS nor DTLS are key exchange mechanisms, and neither accepts raw RSA keys as input, as far as I can see, so this advise looks a bit superficial. (Secondarily, creating a connection with DTLS to exchange keys is quite a lot of overhead, since I only need a key exchange, and isn't neccessarily secure). $\endgroup$ Commented Jul 17, 2013 at 10:28
  • $\begingroup$ Note also that most of my questions are sitll valid even when I use DTLS to exchange key material - I still need to derive cipher and hmac keys. $\endgroup$ Commented Jul 17, 2013 at 10:35
  • $\begingroup$ @MarcLehmann, I suggest editing the question to clarify what you are going to do with the keys. You might be able to use TLS to derive keys and then use those keys for some other purpose, depending upon what you're going to do with them. I'm not seeing why DTLS adds extra overhead over key exchange; the primary overhead in DTLS is (a) the public-key operations, and (b) a few roundtrips. It seems like any key-exchange protocol is going to incur the cost of a few public-key operations and a few roundtrips. $\endgroup$
    – D.W.
    Commented Jul 17, 2013 at 16:38
  • $\begingroup$ For key derivation, use a KDF. See, e.g., Wikipedia, key-derivation, and Deriving Keys for Symmetric Encryption and Authentication $\endgroup$
    – D.W.
    Commented Jul 17, 2013 at 16:41
  • $\begingroup$ As I already wrote in my question, I already use HKDF to derive the keys. $\endgroup$ Commented Jul 18, 2013 at 16:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.