What's the difference between rsa_pss_pss_* and rsa_pss_rsae_* schemes?

Question

I'm trying to understand the difference between rsa_pss_pss_ and rsa_pss_rsae_.

The picture shows the extension of TLS1.3's ClientHello.

I view the OpenSSL code and find the rsa_pss_rsae_* scheme is corresponding to the scheme in Section 8.1 in RFC 8017 but what about rsa_pss_pss_* scheme?

And it's weird that I find some annotation like this:

TODO this does not differentiate between the rsa_pss_pss_* and rsa_pss_rsae_* schemes since we do no have a chain here that lets us look at the key OID in the signing certificate.

What's the matter with this chain and key OID?

In my understanding the signature schemes is only applicable to the type of private key (RSA or EC).

Answer 1

Summary: they aren't different signature schemes. Both specify the use of RSA-PSS. The difference is in how to parse the certificate to find the public key.

RSA-PSS (also spelled RSASSA-PSS and other variants) is a family of short-message signature schemes defined in PKCS#1 (RFC 8017). A short-message signature scheme can only sign a message with a small length limit; normally the payload is a hash of a message that can be arbitrarily long (in pratical terms). RSA-PSS is a family of schemes because it is parametrized by a mask generation function (and a salt length but I won't discuss that here; you can view it as a parameter of each signature rather than a parameter of the signature scheme). The only mask generation function anyone uses is called MGF1, but it is parametrized in turn by a hash function. Thus RSA-PSS-MGF1-SHA-256 is one example of a short-message signature scheme. RSA-PSS-MGF1-SHA-256-SHA-256 and RSA-PSS-MGF1-SHA-256-SHA-512 are two examples of message signature schemes that use the hash-and-sign construction, with RSA-PSS-MGF1-SHA-256 to sign the hash.

rsa_pss_pss_sha256 and rsa_pss_rsae_sha256 both indicate the RSA-PSS-SHA-256 signature scheme, in the same way that rsa_pkcs1_sha256 indicates the RSA-PKCS#1v1.5-SHA-256 signature scheme which uses the RSA-PKCS#1v1.5 short-message signature scheme. The reason there is a pair of values for each PSS-based scheme is that signature algorithm extension in TLS 1.3 encodes a little bit more than a signature scheme.

That extra bit of information is how to find and validate the public key. For the most part, given a TLS version and a ciphersuite, there is only one way to do this: in a ciphersuite that uses certificates (i.e. the “normal” ciphersuites, as opposed to ciphersuites without authentication or using pre-shared keys or raw public keys), you parse the certificate chain, check that the chain of signatures is valid, and check that the root of the chain is acceptable. There is no other method in use in the TLS protocol.

But wait, how do you parse certificates? The TLS protocol only specifies one certificate format, which is X.509. So there's still no need for any extra information. I won't go into detail about X.509, suffice to say that there is a standard way to find the public key and the signature, and both annotated by some information that says what type of key or signature it is. The key or signature type is conveyed by an OID.

Obviously, since there are two standard signature schemes that use RSA keys (RSA-PKCS#1v1.5 and RSA-PSS), they each have their own OID. Less obviously, there are also multiple key types for RSA keys. There's an OID called rsaEncryption which allows the key to be used for any of the standard encryption or (despite the name) signature schemes. And there's an OID called id-RSASSA-PSS which indicates that the key may only be used for PSS signatures. This is explained in RFC 4055. In addition to restricting the use of the key to the RSA-PSS signature scheme family, using id-RSASSA-PSS to identify the RSA key rather than rsaEncryption allows the certificate to indicate which scheme parameters to use (MGF1, its hash, and other rarely-used parameters). The reason to have a distinct key OID is that if you use the rsaEncryption then there is no way to specify that the key must be used for PSS only, and if someone finds something signed with PKCS#1v1.5 then it must be a forgery that takes advantage of a weakness in PKCS#1v1.5. (There is no known weakness of PKCS#1v1.5 when correctly implemented, but one may be discovered one day, and PKCS#1v1.5 verification is difficult to implement correctly.)

So what does this have to do with TLS? Some TLS implementations use an X.509 parser that only recognizes rsaEncryption. So they don't include the rsa_pss_pss_sha256 value when they send the signature algorithm extension, they only include rsa_pss_rsae_sha256 (if they support it), where the rsae part stands for “encoded with rsaEncryption in the certificate”. Others may consider rsaEncryption to be legacy, and they don't include rsa_pss_rsae_sha256, only rsa_pss_pss_sha256 (if they support it), where the second pss stands for “encoded with id-RSASSA-PSS in the certificate”.

Answer 2

As mentioned already, very good question.

What follows is my attempt at tracking down the answer.

I first googled for rsa_pss_rsae_sha256 wireshark and the first result was https://fossies.org/linux/wireshark/epan/dissectors/packet-ssl-utils.c. On line 1285, this links to https://datatracker.ietf.org/doc/html/draft-ietf-tls-tls13-23#section-4.2.3 which in turn contains the following enum on page 46:

enum {
      /* RSASSA-PKCS1-v1_5 algorithms */
      rsa_pkcs1_sha256(0x0401),
      rsa_pkcs1_sha384(0x0501),
      rsa_pkcs1_sha512(0x0601),

      /* ECDSA algorithms */
      ecdsa_secp256r1_sha256(0x0403),
      ecdsa_secp384r1_sha384(0x0503),
      ecdsa_secp521r1_sha512(0x0603),

      /* RSASSA-PSS algorithms with public key OID rsaEncryption */
      rsa_pss_rsae_sha256(0x0804),
      rsa_pss_rsae_sha384(0x0805),
      rsa_pss_rsae_sha512(0x0806),

      /* EdDSA algorithms */
      ed25519(0x0807),
      ed448(0x0808),

      /* RSASSA-PSS algorithms with public key OID RSASSA-PSS */
      rsa_pss_pss_sha256(0x0809),
      rsa_pss_pss_sha384(0x080a),
      rsa_pss_pss_sha512(0x080b),

      /* Legacy algorithms */
      rsa_pkcs1_sha1(0x0201),
      ecdsa_sha1(0x0203),

      /* Reserved Code Points */
      private_use(0xFE00..0xFFFF),
      (0xFFFF)
  } SignatureScheme;

On page 47, we are provided the following descriptions:

RSASSA-PSS RSAE algorithms  Indicates a signature algorithm using
  RSASSA-PSS [RFC8017] with mask generation function 1.  The digest
  used in the mask generation function and the digest being signed
  are both the corresponding hash algorithm as defined in [SHS].
  The length of the salt MUST be equal to the length of the digest
  algorithm.  If the public key is carried in an X.509 certificate,
  it MUST use the rsaEncryption OID [RFC5280].


RSASSA-PSS PSS algorithms  Indicates a signature algorithm using
  RSASSA-PSS [RFC8017] with mask generation function 1.  The digest
  used in the mask generation function and the digest being signed
  are both the corresponding hash algorithm as defined in [SHS].
  The length of the salt MUST be equal to the length of the digest
  algorithm.  If the public key is carried in an X.509 certificate,
  it MUST use the RSASSA-PSS OID [RFC5756].  When used in
  certificate signatures, the algorithm parameters MUST be DER
  encoded.  If the corresponding public key's parameters present,
  then the parameters in the signature MUST be identical to those in
  the public key.

To summarize, it seems it has to do with X.509 OID certificate restrictions.