Cryptography Stack Exchange is a question and answer site for software developers, mathematicians and others interested in cryptography. It only takes a minute to sign up.
Sign up to join this community
To achieve double insurance for authentication, can we sign the GCM final tag with using RSA? with RSA, no one is able to create his own tag since the private key is not known.
However, according to FIPS.186-4, it looks like the input to RSA must be compliant to NIST approved hashing function.
Possible for this GCM + RSA combination?
Not much; signing the GCM tag using RSA is inadequate.
For a start, knowing the AES secret key, it's easy to make a message with any desired content (except in the very end), and an AES-GCM cryptogram that checks, and has any desired tag. Thus any holder of the AES secret key can forge a message that pass verification under the RSA public key of another person, after intercepting a ciphertext signed by that person.
Also, one holding the AES secret key can prepare two very different messages and their AES-GCM encryption sharing the same tag, allowing to spread FUD by showing that the signature of one message also applies to another message.
Further, if RSA-signing the tag was done nevertheless, that must not be with textbook RSA, which is insecure when directly signing small values $m$ such as the 128-bit tag (e.g. when $e$ is so small that $m^e<n\,$, or under the Desmedt-Odlyzko attack for any $e$).
An appropriate way is sign-then-encrypt: signing the plaintext using RSA (e.g. per RSASSA-PSS or RSASSA-PKCS1-v1_5), appending the signature, then using AES-GCM on the whole. This way only the holder of the RSA private key (rather than any holder of the AES-GCM secret key) can make a valid message that pass verification under the corresponding public key; and the signature (which discloses information about what's signed) is not accessible before decryption.
Here I don't recommend encrypt-then-sign, because I do not rule out that the integrity could be vulnerable to a replacement of the original AES secret key.
Generally the RSA key is used to sign the key agreement. The key agreement is used to establish the key used within GCM. Generally the secret key isn't distributed using some kind of out-of-band procedure after all. After that the GCM authentication tag should provide enough security for message integrity and authenticity.
Another issue is that a signature can be replaced by anybody if it is used. For that reason sign-then-encrypt is often used rather than encrypt-then-sign.
It might be interesting to just sign an authentication tag for the encrypt-then-sign in case the use case requires that construction. However, in that case the normal signature generation process could be used. Note that e.g. RSA-PSS uses a hash to pad the hash to the size of the key anyway (using MGF1 internally), and that the modular exponentiation is much compute-heavy than a single hash over 128 bits (which would translate into hashing a single block internally for SHA-2).
