I've used OpenSSL to encrypt some socket communications.
I am however using some functionality from the windows API that prevents me from using OpenSSL's opaque builtin sockets, so I am buffering through their BIO_s_mem interfaces - that requires 4 extra copies for all data passing through (socket->read buffer->read BIO->OpenSSL state machine->write BIO->write buffer->socket, as opposed to socket->OpenSSL state machine->socket).
This is not efficient, especially seen as the goal is (among other things) large file transfers.


As a result I'm thinking I could establish an OpenSSL session, generate some symmetric keys, and use that secure session to share them. I would then open another channel (or just use the underlying socket which I have full control over) to transfer the bulk data using something like AES256-GCM - doing that 'manually' (using OpenSSL crypto libraries) would make it incredibly fast as it would no longer imply the inefficiencies of passing it through the OpenSSL state machine.

Problem, Goal & Question

I'm not a cryptographer. I don't even know what I don't know. All I know is if people can find side-channel attacks based on compression ratio of encrypted data, then any mistakes I make with this would probably be fatal.

I'm aware key generation has to be done very carefully - I've read through this, and I was thinking of using OpenSSL's facilities for the purpose. I'm also aware that AES is by itself secure, but a wrong implementation will easily throw that away.

This does however, on the surface, seem like a manageable challenge. Generate keys -> Run data through AES256-GCM. The upside is big, and this is a personal project that likely won't have the most proficient hackers (or any hackers) trying to mess with it. I just want it secure so that if people start to use it, they're not using something could be easily broken.

So, is there something obvious I'm missing? Something less than obvious?

Extra considerations, perhaps a correction on something wrong I might have said, some resource I should read before the attempt? All is appreciated, but if your contribution is "just don't do it", I ask you to please add some context as to what could go wrong so I get the picture.

  • 3
    $\begingroup$ Why not simply have a look at what TLS is doing here instead of inventing your own and then ask if it is secure? Use something which is proven, don't run your own crypto protocol. Also, I assume that the overhead you envision is not relevant compared to everything else. So better check first that you actually gain relevant performance before trying to implement your own. $\endgroup$ Jan 13 at 8:58
  • $\begingroup$ @SteffenUllrich I'm mostly describing what TLS is doing after the handshake, except they generate the keys based on a bunch of information obtained during it. There may be a reason for that I'm not aware of. And while I never really mentioned performance so much as efficiency - I imagine you're right. Even with the AES instructions, networking and kernel calls might take the bulk (until the data rates get high enough anyway, then DMA probably scales in a way manual copies don't). Regardless I felt it could be worth implementing, and generally a good question for future applications as well. $\endgroup$
    – TrisT
    Jan 13 at 9:43
  • 1
    $\begingroup$ Does this answer your question? Is it secure to use RSA to exchange AES key? $\endgroup$
    – kelalaka
    Jan 13 at 20:36
  • $\begingroup$ @kelalaka mentallurg already linked it in his answer, not my use-case. My use case is using a TLS session to negotiate symmetric keys and then doing the encryption in a separate channel (as opposed to using the TLS session). For the moment it has performance benefits. In the future as I do embedded development, it might become even more useful if I intend on implementing support for hardware crypto units. Doing it in the separate channel wouldn't break or require re-writes when updating openssl (as I wouldn't have to interact with internal APIs). I addressed it in my comments on his answer. $\endgroup$
    – TrisT
    Jan 14 at 4:42

1 Answer 1


Briefly, your approach has following problems that are solved in TLS (see details here.):

  1. It allows man-in-the-middle attacks (TLS doesn't)
  2. It allows replay attacks (TLS doesn't)
  3. It doesn't provide forward secrecy (TLD does)
  4. It doesn't provide key revocation mechanism (TLS does)

Even if one is aware of the these problems and wants to address them in the own solution, a correct implementation of such solutions can require a lot of efforts.

One more potential problem is key distribution. Before any communication you need to provide each 2 parties with a key for their communication. In case one of parties lost the key or the device was compromised, you need again to provide these 2 parties with a new key. If the channel to provide the key is not secure enough, an attacker can intercept the key and can then decrypt the communication.

Some people may feel the usage of TLS "boring" or "not cool" compared to implementing their own solutions. But TLS provides solutions for many security risks.

And, as always for security topics, before making a decision, first evaluate the risks. What will a successful attack cost you and how much efforts and money are you ready to put to prevent it? For instance, if you want to communicate with a bank and a compromised communication can cost you \$100 000 000, then may be you will put a lot of money and time into implementing highly secured connection. But if you want to communicate with a video camera installed in a forest to observe birds nest, then successful attack on this communication will probably doesn't cost you much and it makes no sense to put let say \$10 000 into security.

  • $\begingroup$ I greatly appreciate your answer, but you seem to have misinterpreted what I'm suggesting - that was to establish a TLS session beforehand with which to share the symmetric keys between the 2 parties, and then use those to encrypt the flow of information. I think that would mean MITM is no longer possible, replay attacks too because of the use of GCM (a replayed packet should be erroneous because the counter will have changed on the receiver), I would protect against forward secrecy problems by generating a different key everytime, and key revocation would stay a TLS matter. $\endgroup$
    – TrisT
    Jan 13 at 16:03
  • $\begingroup$ @TrisT: In your question there is not single word about TLS. And if you establish a TLS session, then you have everything: $\endgroup$
    – mentallurg
    Jan 14 at 2:20
  • $\begingroup$ 1) Encryption is made by TLS - why would you need your own encryption? This will add zero security, but will add only obscurity. $\endgroup$
    – mentallurg
    Jan 14 at 2:20
  • $\begingroup$ 2) MITM is prevented by TLS. $\endgroup$
    – mentallurg
    Jan 14 at 2:20
  • $\begingroup$ 3) Replay attacks are prevented by TLS, it is based on sequence numbers - - why would you want to add your additional prevention? $\endgroup$
    – mentallurg
    Jan 14 at 2:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.