Best method to achieve authenticated encryption with openssl

Question

Problem

I want to send a plaintext document between from myself to someone else over an untrusted link. I want to firstly stop third parties from reading the document, I want the receiver to be able to authenticate that any message they receive comes from me. Finally I want the receiver to be able to read the document. I have the ability to give the receiver a key by a trusted channel.

Proposed solution

the document is encrypted with a symmetric cipher. The aes-key I was going to use for that could be turned into a signature. e.g.

openssl pkeyutl -sign -in "aes.key" -inkey privkey.pem -out sig.nature

This signature is not used for authentication (a separate signature based on the hash of the encrypted file is used for that), but it seems to hide the value of the key, which can then be extracted with the public key, e.g. openssl rsautl -verify -inkey pubkey.pem -in sig.nature -pubin >> aeskey.recov

The public key will not be made public, but will be given to the reciever via a trusted channel and secured. I understand however if the public key is ever compromised, then all my encrypted documents can be decrypted.

Is this valid (providing the public key remains private) or is there a better way to do this?

Answer 1

If as you say you

have the ability to give the receiver a key by a trusted channel.

then there is no need to involve any public key cryptography here. Just generate a symmetric key and send that key to your friend over the trusted channel.

Later, once you are ready to send the file, simply encrypt the file using an authenticated encryption mode and send the encrypted file to your friend. They can then use they key to verify the ciphertext's authenticity and decrypt.

You're saying that you specifically want to know how to do that with the OpenSSL commandline tools. This is a bit off-topic here, but nevertheless here we go.

Disclaimer: Do not use anything I write below in any actual application, I make no claim whatsoever that the use of openssl described below is actually secure in practice.

Note, I'll be using bash syntax in the following, I have no idea if it works in other shells.

First thing we need to do is generate keys. We'll be using AES-128, so we will generate 16 random bytes and store them in a key file. (encoded as hex to make our life easier later on.)

openssl rand -hex 16 > enc.key

we will also generate a separate MAC key, I will explain why in a minute:

openssl rand -hex 16 > mac.key

These two files you will give to your friend.

The first idea -- since we want an authenticated encryption mode -- would be to use something like AES-GCM, sadly however, the commandline tool openssl enc will then tell us that

AEAD ciphers not supported by the enc utility

Bummer! And the authors are unwilling to change that. So we will have to do the authentication the hard way. For example with AES-CTR + HMAC. This is why we created a separate MAC key before.

And because the authors of OpenSSL really want you to shoot yourself in the foot: If you actually specify a key, then the openssl enc utility will not create an IV for us, so we will have to do that ourselves as well. So let's get going. We generate an initialization vector

openssl rand -hex 16 > file.iv

and then encrypt our file file.txt using AES-CTR using the generated IV and the key we previously exchanged.

openssl enc -aes-128-ctr -in file.txt -out file.aes -K $(cat enc.key) -iv $(cat file.iv)

compute the HMAC over both the IV and the ciphertext

cat file.iv file.aes | openssl dgst -mac HMAC -macopt hexkey:$(cat mac.key) -out file.mac

and now you can send file.iv,file.aes,file.mac to your friend.

Then again, because the OpenSSL authors really, really want you to shoot yourself in the foot, the openssl dgst utility naturally does not support verification. Instead your friend will have to do verification by hand:

if [ "$(cat file.iv file.aes | openssl dgst -mac HMAC -macopt hexkey:$(cat mac.key))" == "$(cat file.mac)" ]; then
    echo "MAC verification succesful."
    openssl enc -d -aes-128-ctr -in file.aes -out file.txt -K $(cat enc.key) -iv $(cat file.iv); 
else 
    echo "MAC verification failed." 
fi

And if everything went well they should now have the plaintext back.

By now I hope every reader has realized that there's way too much that could go wrong in all of this and has decided not to do encrypted file-transfer by hand.