Summary: they aren't different signature schemes. Both specify the use of RSA-PSS. The difference is in how to parse the certificate to find the public key.
RSA-PSS (also spelled RSASSA-PSS and other variants) is a family of short-message signature schemes defined in PKCS#1 (RFC 8017). A short-message signature scheme can only sign a message with a small length limit; normally the payload is a hash of a message that can be arbitrarily long (in pratical terms). RSA-PSS is a family of schemes because it is parametrized by a mask generation function (and a salt length but I won't discuss that here; you can view it as a parameter of each signature rather than a parameter of the signature scheme). The only mask generation function anyone uses is called MGF1, but it is parametrized in turn by a hash function. Thus RSA-PSS-MGF1-SHA-256 is one example of a short-message signature scheme. RSA-PSS-MGF1-SHA-256-SHA-256 and RSA-PSS-MGF1-SHA-256-SHA-512 are two examples of message signature schemes that use the hash-and-sign construction, with RSA-PSS-MGF1-SHA-256 to sign the hash.
rsa_pss_rsae_sha256 both indicate the RSA-PSS-SHA-256 signature scheme, in the same way that
rsa_pkcs1_sha256 indicates the RSA-PKCS#1v1.5-SHA-256 signature scheme which uses the RSA-PKCS#1v1.5 short-message signature scheme. The reason there is a pair of values for each PSS-based scheme is that signature algorithm extension in TLS 1.3 encodes a little bit more than a signature scheme.
That extra bit of information is how to find and validate the public key. For the most part, given a TLS version and a ciphersuite, there is only one way to do this: in a ciphersuite that uses certificates (i.e. the “normal” ciphersuites, as opposed to ciphersuites without authentication or using pre-shared keys or raw public keys), you parse the certificate chain, check that the chain of signatures is valid, and check that the root of the chain is acceptable. There is no other method in use in the TLS protocol.
But wait, how do you parse certificates? The TLS protocol only specifies one certificate format, which is X.509. So there's still no need for any extra information. I won't go into detail about X.509, suffice to say that there is a standard way to find the public key and the signature, and both annotated by some information that says what type of key
or signature it is. The key or signature type is conveyed by an OID.
Obviously, since there are two standard signature schemes that use RSA keys (RSA-PKCS#1v1.5 and RSA-PSS), they each have their own OID. Less obviously, there are also multiple key types for RSA keys. There's an OID called
rsaEncryption which allows the key to be used for any of the standard encryption or (despite the name) signature schemes. And there's an OID called
id-RSASSA-PSS which indicates that the key may only be used for PSS signatures. This is explained in RFC 4055. In addition to restricting the use of the key to the RSA-PSS signature scheme family, using
id-RSASSA-PSS to identify the RSA key rather than
rsaEncryption allows the certificate to indicate which scheme parameters to use (MGF1, its hash, and other rarely-used parameters). The reason to have a distinct key OID is that if you use the
rsaEncryption then there is no way to specify that the key must be used for PSS only, and if someone finds something signed with PKCS#1v1.5 then it must be a forgery that takes advantage of a weakness in PKCS#1v1.5. (There is no known weakness of PKCS#1v1.5 when correctly implemented, but one may be discovered one day, and PKCS#1v1.5 verification is difficult to implement correctly.)
So what does this have to do with TLS? Some TLS implementations use an X.509 parser that only recognizes
rsaEncryption. So they don't include the
rsa_pss_pss_sha256 value when they send the signature algorithm extension, they only include
rsa_pss_rsae_sha256 (if they support it), where the
rsae part stands for “encoded with
rsaEncryption in the certificate”. Others may consider
rsaEncryption to be legacy, and they don't include
rsa_pss_pss_sha256 (if they support it), where the second
pss stands for “encoded with
id-RSASSA-PSS in the certificate”.