I have made a private messaging program and would like to check that nothing is stupid in my use of cryptography. I am an amateur and nothing is for production. I thank you in advance.
The message exchanges are encrypted end-to-end with AES-OCB. The session key is exchanged as follows:
At the beginning, the private key is loaded and the public key is generated. The server sends its public key to the client. The client verifies the public key of the server with a know_host file. In this file the host addresses are stored hashed (sha256) with their public key (sha256). The client checks that the received public key is the right one, otherwise the sha256 fingerprint of the public key is displayed for verification by the user.
The client generates the 128 bits session key and encrypts it with the server's public key. The client sends to the server: the encrypted session key, its public key, the RSASSA-PSS signature (sha256) of the session key.
The server verifies the identity of the client with its own know_host file. It decrypts the session key. It verifies the signature of the session key with the client's public key.
Both parties are authenticated and have the session key, normal communication begins.