# Best method to achieve authenticated encryption with openssl

Problem

I want to send a plaintext document between from myself to someone else over an untrusted link. I want to firstly stop third parties from reading the document, I want the receiver to be able to authenticate that any message they receive comes from me. Finally I want the receiver to be able to read the document. I have the ability to give the receiver a key by a trusted channel.

Proposed solution

the document is encrypted with a symmetric cipher. The aes-key I was going to use for that could be turned into a signature. e.g.

openssl pkeyutl -sign -in "aes.key" -inkey privkey.pem -out sig.nature


This signature is not used for authentication (a separate signature based on the hash of the encrypted file is used for that), but it seems to hide the value of the key, which can then be extracted with the public key, e.g. openssl rsautl -verify -inkey pubkey.pem -in sig.nature -pubin >> aeskey.recov

The public key will not be made public, but will be given to the reciever via a trusted channel and secured. I understand however if the public key is ever compromised, then all my encrypted documents can be decrypted.

Is this valid (providing the public key remains private) or is there a better way to do this?

• There seems to be some fundamental misunderstanding of public key encryption going on here. A public key cannot be used for decryption. Also I have a very hard time following your first paragraph. Maybe you could try to elaborate a bit on your question to make it clearer. – Maeher Jan 31 '18 at 1:02
• updated, hope it clarifies the question. I understand this isn't the stock method of using public key encryption, thats why i am asking whether it is valid or if there is a major security flaw in this and if there is a better way – rocklegend Jan 31 '18 at 4:12
• I think it would help if you would tell us what your actual goal is, What you currently describe is your approach to achieve something. But it is not really clear what exactly this "something" is. You have a file. I'm following so far. What do you want to do with that file and what is the threat you are trying to protect against? (To be clear I think your question is an example of the XY problem. You're asking us about your solution without explaining the problem you are trying to solve.) – Maeher Jan 31 '18 at 4:23
• updated with a problem statement. – rocklegend Jan 31 '18 at 5:25

If as you say you

have the ability to give the receiver a key by a trusted channel.

then there is no need to involve any public key cryptography here. Just generate a symmetric key and send that key to your friend over the trusted channel.

Later, once you are ready to send the file, simply encrypt the file using an authenticated encryption mode and send the encrypted file to your friend. They can then use they key to verify the ciphertext's authenticity and decrypt.

You're saying that you specifically want to know how to do that with the OpenSSL commandline tools. This is a bit off-topic here, but nevertheless here we go.

Disclaimer: Do not use anything I write below in any actual application, I make no claim whatsoever that the use of openssl described below is actually secure in practice.

Note, I'll be using bash syntax in the following, I have no idea if it works in other shells.

First thing we need to do is generate keys. We'll be using AES-128, so we will generate 16 random bytes and store them in a key file. (encoded as hex to make our life easier later on.)

openssl rand -hex 16 > enc.key


we will also generate a separate MAC key, I will explain why in a minute:

openssl rand -hex 16 > mac.key


These two files you will give to your friend.

The first idea -- since we want an authenticated encryption mode -- would be to use something like AES-GCM, sadly however, the commandline tool openssl enc will then tell us that

AEAD ciphers not supported by the enc utility

Bummer! And the authors are unwilling to change that. So we will have to do the authentication the hard way. For example with AES-CTR + HMAC. This is why we created a separate MAC key before.

And because the authors of OpenSSL really want you to shoot yourself in the foot: If you actually specify a key, then the openssl enc utility will not create an IV for us, so we will have to do that ourselves as well. So let's get going. We generate an initialization vector

openssl rand -hex 16 > file.iv


and then encrypt our file file.txt using AES-CTR using the generated IV and the key we previously exchanged.

openssl enc -aes-128-ctr -in file.txt -out file.aes -K $(cat enc.key) -iv$(cat file.iv)


compute the HMAC over both the IV and the ciphertext

cat file.iv file.aes | openssl dgst -mac HMAC -macopt hexkey:$(cat mac.key) -out file.mac  and now you can send file.iv,file.aes,file.mac to your friend. Then again, because the OpenSSL authors really, really want you to shoot yourself in the foot, the openssl dgst utility naturally does not support verification. Instead your friend will have to do verification by hand: if [ "$(cat file.iv file.aes | openssl dgst -mac HMAC -macopt hexkey:$(cat mac.key))" == "$(cat file.mac)" ]; then
echo "MAC verification succesful."
openssl enc -d -aes-128-ctr -in file.aes -out file.txt -K $(cat enc.key) -iv$(cat file.iv);
else
echo "MAC verification failed."
fi


And if everything went well they should now have the plaintext back.

By now I hope every reader has realized that there's way too much that could go wrong in all of this and has decided not to do encrypted file-transfer by hand.

• I'm sure I'll get a Useless Use of Cat Award for something in there. – Maeher Jan 31 '18 at 7:34
• In my proposal will the signing of the AES key actually encrypt it, or is it only doing some simple operation on the key that is reversible? Or is the better way to encrypt the AES key using a public key provided by the secure recipient? – rocklegend Jan 31 '18 at 8:38
• Signing and encrypting are two completely different concepts. Neither does a signature necessarily hide the message, nor is the message necessarily reconstructible from a signature. So if at all, you should use the recipients public key to encrypt. Though I'll have to ask again: What are you trying to achieve by that? You said that you already have a secure channel to send the key to the recipient. – Maeher Jan 31 '18 at 9:07
• Good point, so whilst it may hide it, this is not guaranteed. – rocklegend Jan 31 '18 at 9:29