Okay to use OpenSSL to encrypt then sign a message?

I am planning on encrypting a message then signing the message so the recipient can authenticate the source. Reading this article has me concerned that I'm doing something wrong.

My current plan is to use OpenSSL to do the following:

• Create a random key
• Use key to encrypt message with a block cypher
• Encrypt key with recipient's public RSA key
• tar the encrypted message and encrypted key
• sign the encrypted package using openssl dgst
• tar the signature and encrypted package for delivery

Is there anything wrong with this approach?

This is being used for transferring firmware for an embedded device. The firmware includes an entire filesystem, including a private key. It needs to be encrypted so the private key is not exposed, and it needs to be authenticated so that a third party cannot install their own firmware and have access to the device (and the private key within).

• Everybody sees who signed the message. – CodesInChaos Jun 10 '12 at 9:13
• Note that a secure digest is not the same as a signature. You can use the digest function as a method to ensure integrity, but you cannot e.g. protect it against alteration by an attacker. – Maarten Bodewes Jun 10 '12 at 17:49
• @owlstead Right - the dgst function has a -sign option to "digitally sign the digest" with a private key. I believe that does protect it against alteration. – Shawn J. Goff Jun 10 '12 at 19:09

I see a few issues with this approach:

• First, since you're signing the ciphertext and sending the signature in plain, anyone who has your public key can verify that you did, in fact, sign that message, even if they won't be able to actually decrypt it. This may or may not be something you want.

• More importantly, anyone who intercepts the message can strip your signature and substitute their own (or, if the protocol allows it, leave the message unsigned).

• Further, if the symmetric cipher used to encrypt the actual message is malleable (as e.g. many common non-authenticating block cipher modes are, at least to some extent), the interceptor might also be able to modify the message before re-signing it, for example replacing an embedded copy of your name or public key with their own.

The obvious solution to these problems is to follow the advice in the article you linked to:

1. Combine (e.g. tar) the plaintext and the public key of the recipient.
2. Sign the combined plaintext and recipient's public key.
3. Combine (e.g. tar) the signature and the signed message.
4. Generate a random key and encrypt the combined message and signature with a symmetric cipher using that key.
5. Encrypt the random key with the recipient's public key.
6. Combine the encrypted message from step 4 and the encrypted key from step 5.

I've followed your example above in keeping the symmetric and asymmetric encryption steps (4 and 5) separate — I'm not familiar with the OpenSSL tools, but many public key crypto implementations provide a way to do both in one step, since it's the only practical way to encrypt large messages with asymmetric crypto. Similarly, the signing step typically involves first hashing the message and then signing the hash, but I haven't explicitly separated those steps since you didn't either.

Ps. It would also be possible (and perhaps preferable) to use key encapsulation (KEM) in steps 4 and 5, if your crypto toolchain supports it. Basically, instead of generating a random symmetric key and then encrypting it with the recipient's public key (which generally involves padding and other complex details), you generate a random chunk of data within the message space of the asymmetric cipher (e.g. a random number less than the modulus, for RSA), encrypt it using the plain "textbook" version of the asymmetric cipher (with no padding or other complications), and also hash it to obtain the key for the symmetric cipher. The recipient then decrypts the random data with their private key and hashes it to obtain the symmetric key. The advantage of KEM is mainly one of simplicity, though, so if you're already working with a toolchain that supports normal asymmetric encryption with padding, but not KEM, then it's probably not worth it.

• I'm going to follow those steps, but I think none of the issues matter in my case. I don't care who knows that the sender signed the message (unless that causes additional problems I don't know about), and the recipient does not trust the message unless it is from the trusted sender. – Shawn J. Goff Jun 10 '12 at 12:38
• @Shawn : $\:$ It's probably not worth the computational and communication cost of fixing this (by signing $\;\;$ the plaintext before encrypting, in addition to signing the ciphertext), but the recipient non-interactively convincing anyone else of what the plaintext was may harm the confidentiality of other messages. – user991 Jul 28 '13 at 10:06