# Constructing a custom protocol using well-vetted algorithms?

For the reasons I cannot go into details, I need to design my own protocol and overall security system which will be used to establish secure data exchange between various devices (most notably desktops of all three major OSes, Android and iOS devices and Raspberry Pi, with the possibility to extend it to other embedded systems). Since I wouldn't dare to do my own crypto algorithms implementation, this implies that I have to rely on tried and tested libraries available for those platforms (i.e. OpenSSL, BouncyCastle, CommonCrypto, …) so I settled with what they all support - RSA for signing/authentication, DHE/ECDHE for session establishment, AES for encryption, PBKDF2 for key derivation and SHA2 for general hashing needs. The key sizes and complexity will depend on the hardware ability of targeted platforms.

Now, I've done a fair bit of implementations of existing protocols and I like to think that I know my way around cryptography and perils that await if not applied properly, however I wouldn't dare to consider myself a cryptography expert and thus I decided to query the hive mind. Of course, once everything completed it will be properly audited and tested, but for now I need to know if I'm on the right track, that I haven't done some glaring omissions and if it is possible to improve it even further.

The system I'm building should both cover the storage on devices and the communication between them and to preserve forward and backward secrecy.

For the storage part I plan to utilize AES in CBC mode, storing the randomly generated IVs with each individual encrypted file. The 'master' key itself will be randomly generated and also AES-encrypted using a key derived from the user's password using PBKDF2, stored along with the salt and iterations used in it. My main concern is the IV part as if generated randomly it might repeat thus compromising the encrypted files if they happen to be similar enough or even the same. It's quite a low chance buy I don't like leaving anything to chance so the question is - given that the IV is public knowledge anyway, is it safe to use incremental progression/timestamp as IV in this case?

The communication part will be a bit trickier as the devices will never communicate directly - instead the data exchange between them will be relied through a third party database to which they periodically connect, which is one of the prime reasons I'm designing my own protocol. If we except this oddity, the data exchange itself should function quite the same as if they were communicating directly, namely the devices will establish a session each time they 'contact' each other and then use the session key for encryption.

To establish a session I plan to use a variation of STS protocol adapted to this environment. The only oddity here is that in some instances only one of the parties involved in communication will posses the RSA key of the other so it won't be fully authenticated - my understanding is that even in this case no successful MITM attack can be performed as at least one of the parties will reject communication if the signature doesn't match. Is that a safe assumption?

Once DH/ECDH keys are exchanged an actual session AES key will be derived from them using SHA2 and the rest of the data exchange will be encrypted using that key, again in CBC mode - until the session expires, when the process repeats. I'm still not clear on how to approach the message integrity and authenticity verification - at first I thought that signing the message with the device's RSA key will be enough, but since it may happen that only one of the parties has the other party's certificate I might add a HMAC alongside with the exchanged messages as well. What would be a recommended, cryptographically secure way to handle that?

• Technically, it is perfectly possible to use SSL/TLS over any kind of odd transport, as long as the transport features reliable serialization of packets. Hence, you might perfectly well use the database server as a link in that communication, and let the two end nodes communicate using SSL/TLS. – Henrick Hellström May 12 '13 at 11:29
• Welcome to Cryptography Stack Exchange. That is quite a long question, and in this current form probably will get only the answers "use TLS for connection, and OpenPGP for storage instead of inventing your own protocols". You'll need to explain why the standard protocols don't fit your needs to get useful responses. Also, I think you should split off the storage part from the connection part, and outline the needs you think you have. – Paŭlo Ebermann May 12 '13 at 11:48
• 1) Cryptography is all about leaving things to chance, just with really small chances. 2) Adding a MAC is essential. HMAC in a encrypt-then-mac contruction is one good solution, dedicated authenticated encryption like AES-GCM another. – CodesInChaos May 12 '13 at 13:19
• Regarding non-random IV in CBC-mode, see the answer to this question. – Roland Smith May 12 '13 at 13:23

• I spotted 3 potential issues in your proposal that are explained there. – I know this is an old question (which probably won’t lead to an accepted answer due to its age) and I sure hope you don’t get me wrong when I write this comment, but it might be helpful for others stumbling upon this Q&A to list those 3 potential issues. (Just to have mentioned them. Adding all the details would surely become too broad when I look at the question.) Thanks in advance for even considering an edit. You don’t have to… I’m just thinking out loud. – e-sushi Aug 29 '16 at 14:59