For my personal research, I have to create a messaging app, but security is only important part of application, security from MITM (man in the middle attack), at device end, and at server level. (note: this will not end to end encryption, as I want to backup messages on server for cross-device access like telegram).
I created the following model:
Server creates RSA key pair first time at installation and keeps private key in a file on server, and broadcast public key with its clients, (A, B) A&B also create RSA key pair at first-time login, and stores private key at device level and sends public key to server. A sending message to B: create a random salt_key encrypt message with this salt_key (AES) encrypt salt_key with server's public key (RSA) send RSA encrypted key, and AES encrypted message to server server can decrypt salt_key with RSA private key and then message server will do same as A did, but with B's public key and B will decrypt salt_key with private key then decrypt message.
I Will also add "signature" for sender verification, but my concern is SERVER because if server compromised everything the-end, but if i choose end to end encryption, how to achieve my message backup goal and cross device, access.