Designing a secure IM protocol

Question

I haven't been programming much lately, so I figured I would make an encrypted instant-messaging program (read: Skype clone) with groups to hone my skills. That's not important though, I can do that. The problem is that I know almost nothing about cryptography, so I figured I'd ask here so I don't discover down the road that my way of doing crypto is all wrong and I have to remake everything.

Here's my plan:

• When a client connects, it generates a keypair (or loads one from disk) and sends the public key off to the server.
• The server broadcasts the public keys of everyone in a group chat to everyone else.
• When a client sends a message, it encrypts (with NaCl's crypto_box) the message many times, once with each public key. Then it sends all of them off to the server, including information about which encrypted message goes to which client.
• The server simply forwards the messages to the recipients, and the recipient decrypts it with his own private key and the sender's public key (again with NaCl's crypto_box).

There will of course be authentication, so someone can't just send their public key to the server, pretending to be someone else.

I have no idea how secure that method is, or if it can even work. If this is not optimal, how would you do it?

Answer 1

I think the very first question before diving into designing the protocol would be to ask yourself what kind of secure communication do you want? Namely, there are two options to consider when it comes to communication between the IM users:

1. For the client-server secure communication, there are two directions to consider:

   1. From “sender to server”, you would have to encrypt the message with the public key of the server (assuming using public key encryption).

   2. From “sender to recipients”, the server would encrypt each message with the respective recipients' public key (assuming using public key encryption). However, if you are consider using symmetric encryption, a good reference for protocol would be the SSL/TLS protocol that is used widely.

2. For the end-to-end secure communication, it would be difficult to implement for group message but simpler for individual message. It means that the sender will encrypt the message with a predefined encryption key, and only people who know the encryption key is able to decrypt the message. Sharing the predefined encryption with a group might compromise on the overall security.

Once decided on the framework type, you can add additional information to the payload (i.e. message) to ensure there is authentication, non-repudiation, data integrity, etc. A good reference would be the SSL/TLS protocol.