I recently came across this (PDF) journal article, which describes what it calls "A New Design Of A Hybrid Encryption Algorithm" (which I must say, doesn't sound very novel, even in 2014!):

  1. An AES key K is chosen
  2. Encrypt message M using AES and key K

    eM = AES-encryption(M)

  3. AES key K is encrypted using RSA with the public key of the receiver

    eK = RSA-encryption (K)

  4. The cipher text eM is fed to SHA1, which generates a 160-bit message digest

    mD = SHA1 (eM)

  5. The message digest is signed using RSA with the private key of the sender

    DS = RSA-sign (mD)

  6. The encrypted message eM, digital signature DS and AES encrypted key eK are transmitted to the user over a network

So, basically standard AES-RSA hybrid encryption, but with the additional of an RSA signature (a signed message digest) that is intended to provide message authentication.

My question is whether signing the ciphertext actually provides any extra security (i.e. integrity or authentication) - doesn't encrypting the AES key with the receiver's RSA public key effectively provide message authentication?

  • $\begingroup$ the encryption provides recipient authentication, not sender authentication, that is anybody could encrypt a message for Bob claiming to be "Alice" without signatures involved. However, the given approach is kinda bad, because it allows anybody to find out the source of the message, ideally you want to sign the actual message, encrypt that and then MAC the ciphertext. $\endgroup$
    – SEJPM
    Apr 20 '18 at 10:21
  • $\begingroup$ @SEJPM Are you saying you should hash the message (plaintext or ciphertext?) and RSA sign the hash, then encrypt the signature and append it to the ciphertext? What is the benefit of encrypting the signature? $\endgroup$
    – Cocowalla
    Apr 20 '18 at 10:30

My question is whether signing the ciphertext actually provides any extra security (i.e. integrity or authentication) - doesn't encrypting the AES key with the receiver's RSA public key effectively provide message authentication?

With asymmetric encryption, you need to consider two different kinds of authentication of the message:

  • Sender Authentication. That is, the fact that the recipient can be sure who sent a given message. For this you actually need signatures (or MACs with pre-shared keys), as without a signature, an attacker could just intercept a ciphertext, drop it, and encrypt a different message and send that.
  • Recipient Authentication. That is, the fact that you can actually, as a sender, be sure that that the message reaches the intended recipient. You essentially get this for free with hybrid encryption.

Now to a side-note: The given mechanism signs the ciphertext. It is preferable to sign the plaintext, append the signature and encrypt the combination. For details, please see this question: "Should we sign-then-encrypt, or encrypt-then-sign?".

As to how ideally do things, let's go through this: Let $H$ be a random oracle (e.g. instantiated by SHAKE128) with output size 224 bit. Let $(n,e)$ be a RSA public key, let $\operatorname{sign}_{sk}(m)$ be any EUF-CMA secure signature scheme (eg ECDSA). And finally let $E_{K\parallel IV}(M)$ denote AES-128-GCM encryption using $K$ and $IV$, returning $(c,\tau)$. Then the following is the "proper" way to encrypt and sign a message $m$:

  1. $r\xleftarrow{\$}\{0,...,n\}$, that is, sample an integer smaller than $n$, uniformly at random and call it $r$.
  2. $c_1\gets r^e\bmod n$, that is, text-book RSA encrypt the random value and call the result $c_1$. This is secure because we are doing RSA-KEM here.
  3. $m'\gets (m,\sigma=\operatorname{sign}_{sk}(m))$, that is, sign the message, call the signature $\sigma$ and pair it with the message (appending suffices if $\sigma$ has a known length) and call the result $m'$.
  4. $k\parallel IV\gets H(r)$, that is, apply the key derivation function / random oracle / hash function to get the key and IV for the symmetric encryption.
  5. $c_2\gets E_{k\parallel IV}(m')$, that is encrypt and symmetrically authenticate the message.
  6. return $(c_1,c_2)$, that is, return a pairing of $c_1$ and the encrypted message. Simple concatenation suffices if $c_1$ is left-zero padded to the same length of $n$.
  • $\begingroup$ Would you mind expanding on your last paragraph on how you think it should be done? Since there is both RSA and AES encryption at play, and you don't mention how to handle the AES IV and key, it would be great to get a complete step-by-step to make it clear :) $\endgroup$
    – Cocowalla
    Apr 20 '18 at 15:34
  • $\begingroup$ @Cocowalla I have done so (and I think in more detail than the paper even ;) $\endgroup$
    – SEJPM
    Apr 20 '18 at 15:52
  • $\begingroup$ Haha, wonderful, thanks! Only thing I think is missing is including the IV in the filal concatenation? (IV, c1, c2) $\endgroup$
    – Cocowalla
    Apr 20 '18 at 15:59
  • $\begingroup$ @Cocowalla the IV is derived from the random value, ie $c_1$ upon decryption along with the key. $\endgroup$
    – SEJPM
    Apr 20 '18 at 16:07
  • $\begingroup$ Ah, sorry, I missed that! $\endgroup$
    – Cocowalla
    Apr 20 '18 at 17:20

Yes, signing the ciphertext does provide extra security, and yes, that would be integrity and authenticity of the message.

Encrypting with a public key does not provide integrity / authenticity against active attacks. That's kind of obvious: an adversary can simply encrypt another message with the public key - it is public after all.

The scheme however uses asymmetric encrypt-then-sign. This is dangerous as an adversary could strip off the signature to replace it with its own. That way it looks like the adversary has signed / send the message. However, this is only possible if the public key of the adversary is trusted within the system. More details are (indeed) here.

It will also be impossible to verify the message afterwards (unless you keep the ciphertext message as well, and make sure that the plaintext message and ciphertext message are irreversibly linked).

So when used correctly this kind of scheme provides integrity / authenticity. But it also has known limitations; that is: known to the cryptographic community, not the authors.

These kind of papers are used by the writers to get their degrees or progress within their studies. They provide zero insight and contain clear issues:

  • no mode of operation is defined for the symmetric cipher, no IV handling is performed;
  • the padding modes of RSA are not specified, not for encryption nor for signature generation ... there is only a limited description of the textbook RSA operation;
  • the hashing is not thought of to be part of the signature generation function;
  • the use of SHA-1 and the references to MD5 and blowfish, years and years after SHA-2 has been standardized.

And quite obviously there is no security analysis at all, which makes the entire paper a waste of bits and - worse - time.

The given "protocol" (and the quotes are needed here) could easily be extended with a message nonce and sender/recipient, making it more secure.

  • 3
    $\begingroup$ Asking about the paper however isn't a waste of time :) $\endgroup$
    – Maarten Bodewes
    Apr 20 '18 at 11:04

Your design is exploitable to MITM (man-in-the-middle) attack, an attacker can intercept your message and deliver "personal tampered message" encrypted with receiver's public key.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.