I designed a custom handshake between 2 entities $\rm C$ (client) and $\rm S$ (server) which should meet the following requirements:


The handshake should make:

  • $\rm S$ know who is $\rm C$,
  • $\rm C$ make sure that he is talking to $\rm S$,

... without letting anyone who is listening the communication line know:

  • what is $\rm S$'s public key,
  • what is $\rm C$'s public key,
  • whether or not a specific $\rm C$ with a well known public key is in the contact list of a specific $\rm S$ with a well known public key.

My handshake:

  1. $\rm C$ opens a connection to $\rm S$.

  2. $\rm C$ sends a random cipher's symmetric key $K_1$ to $\rm S$, encrypted using $\rm S$'s public key. Then $\rm C$ uses $K_1$ in a symmetric cipher to encrypt all the following data:

    • timestamp,
    • 2 first bytes of $\rm C$'s public key.
    • $\rm C$'s signature, where the hash is computed from:

      • $K_1$,
      • timestamp,
      • $\rm S$'s public key.
  3. $\rm S$ receives it and decodes it using his private key. If the timestamp is too far from now or If the signature cannot be verified then close the connection. Otherwise the message is considered valid, then $\rm S$ sends back the following concatenated message to $\rm C$, encrypted using $\rm C$'s public key:

    • a random cipher's symmetric key $K_2$.
    • a hash of $\rm C$'s signature from the previously received message.
  4. $\rm C$ takes $K_2$ and verifies that the hash is from the signature that he previously sent. If it doesn't match he closes the connection.

  5. Both $\rm C$ and $\rm S$ compute $K$ by combining $K_1$ and $K_2$ together. They now have a shared secret that they will use to encrypt and decrypt any further message via a symmetric encryption algorithm.


  • $\rm C$ and $\rm S$ know each other's public key.
  • the handshake description is also available on this page.

Question: I would like to know if my handshake meets its requirements.


  • The asymmetric encryption algorithm used is RSA with 4096 bits keys.
  • The symmetric encryption algorithm used is AES-256.
  • The shared secret algorithm used to combine $K_1$ and $K_2$ is Diffie-Hellman.
  • 1
    $\begingroup$ I suppose the point of the "first 2 bytes of C's public key" is to serve as a semi-unique user ID, so that S doesn't need to try verifying the signature with every public key they know? $\endgroup$ Apr 7, 2012 at 12:16
  • 1
    $\begingroup$ I'd also note that this sounds like a good opportunity to use key encapsulation. You'd have to modify step 3 slightly (to encrypt the hash with $K_2$ instead of directly with C's public key), but then you wouldn't have to worry about things like padding schemes. $\endgroup$ Apr 7, 2012 at 12:27
  • $\begingroup$ @IlmariKaronen: You guessed correctly, the 2 bytes are for this usage. For your second note, you are right, and in my implementation I used a SHA-256 hash of K2 as the key of a AES-256 cipher to encode the hash of C's signature. I just forgot to write it in my documentation - sorry. $\endgroup$
    – Vincent
    Apr 7, 2012 at 13:01

2 Answers 2


No, the handshake does not meet your requirements.

Nowhere do you state that the asymmetric encryption algorithm used does or must have the property that it not be possible to extract the particular public key used to encrypt a particular message. This doesn't violate any of the basic rules of an asymmetric encryption algorithm which only requires that that messages encrypted with the public key cannot be decrypted without the private key.

So, assume the implementation uses an algorithm that does have this property. An interceptor can then receive the encrypted key sent in the first step and extract S's public key from it. Oops.

You either need to specify the particular algorithms and data structures used or you need to impose appropriate constraints on the algorithms chosen to ensure the security requirements are met.

  • 1
    $\begingroup$ You are right. I will edit my question to specify them. $\endgroup$
    – Vincent
    Apr 7, 2012 at 14:05
  • $\begingroup$ And now, does it meet the requirements? $\endgroup$
    – Vincent
    Apr 8, 2012 at 11:34

First off just use SSL. It has a client authentication scheme built in and is way better tested and thought out than things that anyone can come up with.

So there are a few things that just don't add up.

First you use diffie hellman to have two public keys that want to decide on a private key. In this instance you have two shared secrets you want to compute a third master secret. You could hash them, then make each one it's own DH key in some group and then combine them but it's way overkill for no reason. For your purposes probably use HMAC(K1,K2)

Next, If everythings a 4096 bit key, then they'll usually start out with the same number. 4096 bit means its around 2**4096 as such, your identifier is pretty much useless. I'd suggest hashing the key, and using that as an identifier.

Third there's an actual signature operation in public key cryptography. Use it. You also arguably have message confusion unless you fix the length of the timestamp.

Fourth- You need an iv somewhere. Else I can easily see who it is since the timestamp may not actually influence the public key identifier. As such an attacker listening on the wire can figure things out about who's talking.

  • $\begingroup$ I didn't use SSL because it seems that the server is giving away its public key during the handshake. 1) I don't understand which 2 shared secrets you mean. 2) I don't understand why the first 16 bits of those keys will be the same, I didn't experiment that issue. 3) I am using the standard RSA signature, and the timestamp has a fixed size. 4) I don't understand which IV you mean. $\endgroup$
    – Vincent
    Apr 8, 2012 at 18:04
  • $\begingroup$ 1) Now I see which 2 shared secrets you meant. I will consider your suggestion, thx. 4) Now I see what you mean. I think I am using a hash of K1/K2 as IV. $\endgroup$
    – Vincent
    Apr 8, 2012 at 18:29
  • $\begingroup$ I apologize i read it as the signature is the hash of those items not the signature is Sign(H(K1||Time||SPub)). And the whole world can know your public key that doesn't really help them. $\endgroup$
    – Ben
    Apr 8, 2012 at 18:39
  • $\begingroup$ In my application, I don't want Eve to know who is talking to who. $\endgroup$
    – Vincent
    Apr 8, 2012 at 18:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.