I haven't been programming much lately, so I figured I would make an encrypted instant-messaging program (read: Skype clone) with groups to hone my skills. That's not important though, I can do that. The problem is that I know almost nothing about cryptography, so I figured I'd ask here so I don't discover down the road that my way of doing crypto is all wrong and I have to remake everything.

Here's my plan:

  • When a client connects, it generates a keypair (or loads one from disk) and sends the public key off to the server.
  • The server broadcasts the public keys of everyone in a group chat to everyone else.
  • When a client sends a message, it encrypts (with NaCl's crypto_box) the message many times, once with each public key. Then it sends all of them off to the server, including information about which encrypted message goes to which client.
  • The server simply forwards the messages to the recipients, and the recipient decrypts it with his own private key and the sender's public key (again with NaCl's crypto_box).

There will of course be authentication, so someone can't just send their public key to the server, pretending to be someone else.

I have no idea how secure that method is, or if it can even work. If this is not optimal, how would you do it?

  • $\begingroup$ One obvious issue is that you can send a different message to each recipient with that scheme. $\endgroup$ Feb 27, 2014 at 16:46
  • $\begingroup$ @CodesInChaos Why is that a problem? $\endgroup$ Feb 27, 2014 at 16:58
  • $\begingroup$ Hmmm… the authorization part is a bit “undescribed”. ;) How and where in that protocol are you planning to implement authorization? I'm asking because it's an important part of what you're planning to do… and doing it wrong might break your protocol quicker than anything else. (Without that information, answers would have to guess if authorization would be able to kill Schrödinger’s cat or not – and when it comes to crypto, the worst is to be assumed.) $\endgroup$
    – e-sushi
    Feb 27, 2014 at 17:06
  • $\begingroup$ @e-sushi I'm planning to do a simple username-password thing. Right username/password, and the server accepts it. Wrong, and you get kicked off the socket. $\endgroup$ Feb 27, 2014 at 17:11
  • 1
    $\begingroup$ @e-sushi Sorry, English is not my first language. Once a client connects, it'll have 5 seconds to give a valid username/password until it gets kicked off. If it does give a valid username/password, it'll be given a session key (stored in a database), which it'll use in every other API call. If it gives an invalid session key to any of the other calls, it either gets kicked off or simply gets an error message (haven't really decided yet). $\endgroup$ Feb 27, 2014 at 17:26

1 Answer 1


I think the very first question before diving into designing the protocol would be to ask yourself what kind of secure communication do you want? Namely, there are two options to consider when it comes to communication between the IM users:

  1. For the client-server secure communication, there are two directions to consider:

    1. From “sender to server”, you would have to encrypt the message with the public key of the server (assuming using public key encryption).

    2. From “sender to recipients”, the server would encrypt each message with the respective recipients' public key (assuming using public key encryption). However, if you are consider using symmetric encryption, a good reference for protocol would be the SSL/TLS protocol that is used widely.

  2. For the end-to-end secure communication, it would be difficult to implement for group message but simpler for individual message. It means that the sender will encrypt the message with a predefined encryption key, and only people who know the encryption key is able to decrypt the message. Sharing the predefined encryption with a group might compromise on the overall security.

Once decided on the framework type, you can add additional information to the payload (i.e. message) to ensure there is authentication, non-repudiation, data integrity, etc. A good reference would be the SSL/TLS protocol.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.