# How are the instantiations of RSAES-OAEP and SHA*WithRSAEncryption different in practice?

For the spare-time project I had been working on, I'm evaluating the PKCS#1 padded RSA schemes for implementation.

For PKCS#1 v1.5, encryption doesn't seem to require a hash function, and the signature doesn't need additional mask-generating function (MGF) beyond a digest algorithm for hashing the message.

For PKCS#1 v2.x, both encryption and signature are instantiated with a MGF, a hash function, and an optional label (which has no specified use currently). The MGF which, in turn, is instantiated with another hash function using the (ad hoc) MGF1 construction.

In my opinion, a composite cryptosystem should have as few instantiation parameters as possible, so as to ease implementation burdon and promote interoperability; only parameters necessary for cryptographic agility should be inserted into the design.

Which is why, it comes to me as a suprise, that the hash that instantiates the MGF can be different from that used for hashing the input.

My question here are:

• are there existing recommendations (e.g. CMS, PKIX) on the use and instantiation of PKCS#1 v2.x RSAES-OAEP and RSASSA-PSS, where the selection of hash functions are specified?
• Are there assigned IDs (from IANA or similar organizations)?
• Additional points goes to treatment of NIST Draft FIPS 186-5, where SHAKE-{128,256} are to be approved for use as MGF.
• PKCS#1 recommends using the same hash functions (and mandates using the same hash function to hash the message and to hash the hash, it's only the MGF which is allowed to be different), and in my experience this is mostly followed in practice, but I've seen systems that got stuck on SHA-1 for the MGF despite having moved to SHA-2 for message hashing. I've never seen anything other than MGF1 for the MGF (aside from SHAKE, but if there are early adopters I haven't seen them yet: everything I've seen using SHAKE pairs it with ECC). Aug 17, 2021 at 7:31

• are there existing recommendations (e.g. CMS, PKIX) on the use and instantiation of PKCS#1 v2.x RSAES-OAEP and RSASSA-PSS, where the selection of hash functions are specified?

Both CMS and PKIX have recommendation on this. Both CMS's RFC-4056 and PKIX's RFC-4055 recommends using the same hash for the message and MGF.

Side note: the S/MIME and PKIX are concluded work groups at IETF, their successor LAMPS is doing the maintanance chores.

• Are there assigned IDs (from IANA or similar organizations)?

ASN.1 OIDs don't usually need to go through IANA.

RFC-3560 go as far as to list the octet values for the algorithm parameter ASN.1 DER encoding, verbatim.

• Additional points goes to treatment of NIST Draft FIPS 186-5, where SHAKE-{128,256} are to be approved for use as MGF.

NIST, like any other official government organs, had always maintained vendor-neutrality position, and disclaim any endorsement of commercial products mentioned in their official publications.

The IETF RFCs-{8692,8702} specifying SHAKE for use with RSA (and CMS and PKIX in general), had listed Cisco Systems as co-editors. So this is just a guess, maybe Cisco is introducing products that may gain from the hardware efficiency of Keccak permutation and SHAKE sponge parameters, when used as random oracle in the RSA algorithm.

The later is the modern near-substitute of the former.

First and foremost, it's essentially impossible to make a library implementing RSAES-PKCS1-v1_5 decryption that insures security against side-channel attacks. Many attempts to get application-level security have failed: an adversary able to make a modest number of queries to a device performing decryption and observe it's behavior (error code, packet size, timing, disk accesses, cache, power supply sound…) often succeeds [that is manages to decrypt one message they intercepted, or sign one message if the key is dual use]. Bleichenbacher's CCA attack has so many variations that it's hard to keep track. Contrast with RSAES-OAEP: a limited amount of care in a library implementation prevents the equivalent attack.

Also, RSAES-PKCS1-v1_5 is defined to allow down to 64-bit randomness, which is not enough by modern standards to robustly prevent testing an exact guess of plaintext. The only way to fix this is to prevent encryption of messages too close to the maximum size allowed.

RSAES-OAEP has a proof of security (under the assumption the RSA problem is hard, and the hash can be modeled as a pseudo-random function, and the implementation has no side channel). RSAES-PKCS1-v1_5 does not.

One downside of RSAES-OAEP is that it allows smaller messages for the same modulus, but that's seldom an issue in practice, for when message size grows we use hybrid encryption.

existing recommendations (e.g. CMS, PKIX) on the use and instantiation of RSAES-OAEP and RSASSA-PSS

As pointed in comment to the question, the current best practice is to use MGF1 with the same hash as for the rest of the construction. SHA-256 is common, SHA-512 would be best. The most common deviation from that is use of SHA-1 in MGF1 for RSAES-OAEP, because the Java API makes it an easy mistake (see this). That's still secure as far as we know.

For RSASSA-PSS, a common (but not the best) practice is to leave the salt field empty, that is use $$sLen=0$$ in EMSA-PSS.

Using SHAKE-{128,256} as MGF would be sound in theory, but has not made it to practice in applications I know, and might never do.

Are there assigned IDs (from IANA or similar organizations)?

There are OIDs for RSAES-OAEP, RSASSA-PSS, MGF1, SHA-256 and SHA-512. Now how should these OIDs be used in combination in e.g. an X.509 certificate, and is there something that will keep on working if that's attempted? I pass.

• For most people PKIX is as good as X.509 and PKIX params for OAEP and PSS are in RFC 4055. TLS1.3 (RFC 8446 4.2.3) uses PSS signatures, specifying MGF-hash = message-hash and saltlen = digestlen, and the certificate must contain either the original OID (rsaEncryption for historical reasons even though signature isn't encryption) or the PSS OID. TLS1.3 is pretty widely implemented, though I don't know how many sites use RSA-PSS versus ECDSA or EdDSA -- maybe time for EFF to do another survey? Aug 18, 2021 at 1:51