Can AES-CBC be safely used in combination with RSA, or must it be CBC-MAC?

Question

The plan is to encrypt files using an AES-256 in CBC mode secret key. The secret key itself will be encrypted with a RSA 2048-bit public key.

Is using AES-CBC sufficiently secure if used in conjunction with RSA?

Or must the files be authenticated with AES as well using CBC-MAC, in addition to encrypting the symmetric key with RSA and the data with CBC?

It may be assumed that integrity is required for both the symmetric key and the data.

Answer 1

It depends on the system if RSA + CBC mode is secure. It could be but better options are available.

CBC mode, if correctly applied, can certainly provide confidentiality of the message. In that case however you must make sure that padding oracles and - more generically - plaintext oracles do not apply to CBC mode. If RSA with PKCS#1 v1.5 padding is used - often the default - then padding oracles may also apply to RSA (Bleichenbacher attack).

CBC mode doesn't supply integrity / authenticity of the plaintext message. Although an attacker cannot flip each individual bit (like with CTR mode), an attacker may still garble plaintext blocks by changing, moving or inserting/deleting blocks of plaintext. Then again, as anybody with the public key can perform RSA encryption. So an attacker may as well generate a complete message and encrypt that. The addition of CBC-MAC or authenticated modes of encryption like GCM encryption won't help you achieve authenticity / integrity of the message.

If you want to achieve integrity / authenticity then it is recommended to first sign the message with a private key and then encrypt both using hybrid encryption. If you would not encrypt the signature then you would loose the anonymity property and you could leak the hash over the message. You would of course need a key pair generated by the sender for that, and the receiver should trust the public key of the sender.

CBC mode would still be vulnerable against padding oracles this way so although you could use it then you are better off using e.g. AES in counter mode (making sure that the IV is properly random or at least unique).

In the end I'd recommend something like RSA-OAEP (or RSA-KEM if you can find a good implementation) with CTR mode. If you need integrity and authenticity then perform RSA-PSS with a SHA-2 512 or SHA-3 512 bit hash and encrypt both the message and the signature.

---

Notes:

• Performing encrypt-then-sign - where the ciphertext is signed - is more dangerous as anybody could replace the signature. This is a problem if the keys of the attacker are trusted by the system.
• Your security parameters aren't all that well balanced; RSA-2048 has a strength of about 112 bits (according to NIST, 2017). Using that with AES-256 with a strength of 256 bits is fooling yourself. I'd use a combination of AES-128 with a 3072 bits key or use AES256 and a much higher key size such as 8192 bits or beyond. If you want optimal security you could think of switching to Elliptic Curves or an asymmetric encryption algorithm that delivers post quantum cryptography.

Answer 2

There is a weak security property that RSAES-OAEP + AES-CCM (or any KEM + authenticated encryption) provides that RSAES-OAEP + AES-CBC (or any KEM + unauthenticated encryption) does not: Without authentication, a MITM on the channel from sender to receiver can make a small tweak to a message that they otherwise don't know much about; with authentication, the MITM can only replace the message wholesale.

Generally, it is better to just use authenticated encryption everywhere by default unless you have an overwhelming performance constraint and a security justification for avoiding authentication. This is why ‘DEM’ as the modern approach to public-key encryption—which is how all the NIST PQCRYPTO submissions work, for instance—is defined to be the composition of KEM with authenticated encryption.

Don't think of it as AES in CBC mode. Think of AES-CBC as a tool with a security contract—and a confusing one, at that; you are better off with the security contracts of authenticated encryption tools like AES-GCM (also detailed at that link) or AES-CCM or, better, NaCl crypto_secretbox_xsalsa20poly1305.

Finally, if you want public-key authenticated encryption, where the sender and receiver both share their public keys in the telephone book, and the sender sends a message that only the receiver can open and that the receiver can verify came from the sender, then you should just use a public-key authenticated encryption scheme like NaCl crypto_box_curve25519xsalsa20poly1305.

Note that public-key authenticated encryption is different from the composition of public-key signature and public-key anonymous encryption: first, generic composition of them is tricky; second, a public-key signature is verifiable by anyone, not just the receiver, which means a malicious receiver can present the signed to a third party who also knows the sender's public key. With mere public-key authenticated encryption, the third party has no cryptographic assurance the receiver didn't just forge the putative message, because only the receiver can verify messages.