Q: I am using CAPI Engine in OpenSSL and I did some test. When I use TLS 1.0 or 1.1, during handshake and RSA signing, PKCS padding is chosen. When I use TLS 1.2, RSA signing uses PSS padding.
Funny, a quick lookup of TLS 1.2 contains the following:
Note that there are certificates that use algorithms and/or algorithm
combinations that cannot be currently used with TLS. For example, a
certificate with RSASSA-PSS signature key (id-RSASSA-PSS OID in
SubjectPublicKeyInfo) cannot be used because TLS defines no
corresponding signature algorithm.
As I didn't find any RFC's that counteract this, it seems impossible that TLS 1.2 uses PSS for authentication at all.
Now, on the other hand TLS 1.3 specifies that you should use PSS, which is not that surprising as TLS 1.3 very deliberately uses up to date cryptography at the cost of backwards compatibility. The TLS 1.3 specification does specify in section 4.2.3 that a 1.3-capable implementation negotiating 1.2 must support PSS signatures. If the peer supports 1.3, there should be no good reason not to negotiate 1.3 though, and if it only supports 1.2 it should not allow PSS, as there is no way to indicate that padding format1.
1 Choosing the padding algorithm can be performed by guessing the signature format, but PKCS#1 explicitly requires you to define the configuration beforehand, rather than looking into the signature format itself, as that may leave you prone to downgrading attacks on the signature algorithm itself.
For TLS 1.0 a very specific structure with both SHA-1 and MD5 is used. This standard predates SHA-2 and therefore it uses a more secure construct using two hashes that are separately pretty insecure. It also may use RSA encryption using PKCS#1 v1.5 padding as PKCS#1 v2.0 hadn't been introduced either. I don't see any changes for TLS 1.1 for RSA signing / encryption. Note that TLS 1.1 and 1.2 were mainly incremental updates of TLS 1.0 rather than the significant rewrite that TLS 1.3 is.
So basically, the use of RSA padding in TLS is directly tied to the standard itself. OpenSSL will try and follow that standard.
It might be that if you have a certificate that identifies the public key algorithm as PSS, see here. In that case an implementation may decide to go with that single choice, even though it is prohibited by the TLS 1.2 standard - I guess the correct response would be to throw an error unless the connection is downgraded from TLS 1.3.