I have made a private messaging program and would like to check that nothing is stupid in my use of cryptography. I am an amateur and nothing is for production. I thank you in advance.

The message exchanges are encrypted end-to-end with AES-OCB. The session key is exchanged as follows:

At the beginning, the private key is loaded and the public key is generated. The server sends its public key to the client. The client verifies the public key of the server with a know_host file. In this file the host addresses are stored hashed (sha256) with their public key (sha256). The client checks that the received public key is the right one, otherwise the sha256 fingerprint of the public key is displayed for verification by the user.

The client generates the 128 bits session key and encrypts it with the server's public key. The client sends to the server: the encrypted session key, its public key, the RSASSA-PSS signature (sha256) of the session key.

The server verifies the identity of the client with its own know_host file. It decrypts the session key. It verifies the signature of the session key with the client's public key.

Both parties are authenticated and have the session key, normal communication begins.


1 Answer 1

  1. Your scheme has no forward secrecy. If an attacker obtains the server private key someday, all past sessions can be decrypted: Session keys can be decrypted and thus the corresponding sessions can be decrypted. See details here.

  2. You scheme is prone to replay attack. If the attacker intercepts the traffic and knows the meaning of particular part of the encrypted data, then the attacker can send these data again. If it is done within the same session, then the same encryption key is still used, and it will be impossible to detect if the message was really send by the other side or if it was sent by the attacker. Depending on the message semantic, it can be dangerous. See details here.

  • $\begingroup$ Also the method the client uses to encrypt the session key wasn't specified. If it's RSAES-OAEP, that's OK, but just about anything else is going to have some caveats or be downright unsafe. And it's arguably better to use RSA-KEM or ECDH to generate a shared secret rather than encrypt and send one. $\endgroup$ Jan 7, 2022 at 3:14
  • $\begingroup$ @SAIPeregrinus: Sure. And SSH tunnel or TLS would be better than self made protocol. $\endgroup$
    – mentallurg
    Jan 7, 2022 at 3:17
  • $\begingroup$ Agreed. Or at least a Noise protocol, if SSH or TLS (or Wireguard) don't work for whatever reason. $\endgroup$ Jan 7, 2022 at 3:36
  • $\begingroup$ For the encryption of the session key I use RSAES-OAEP. If I use ECDH for the session key exchange, will the "no forward secrecy" problem be solved? $\endgroup$
    – Laurent57
    Jan 7, 2022 at 18:27
  • $\begingroup$ @Laurent57: Not quite. It should be ECDHE, ephemeral ECDH. Then you will have forward secrecy. $\endgroup$
    – mentallurg
    Jan 7, 2022 at 21:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.