Q: I am using CAPI Engine in OpenSSL and I did some test. When I use TLS 1.0 or 1.1, during handshake and RSA signing, PKCS padding is chosen. When I use TLS 1.2, RSA signing uses PSS padding.
Funny, a quick lookup of TLS 1.2 contains the following:
Note that there are certificates that use algorithms and/or algorithm
combinations that cannot be currently used with TLS. For example, a
certificate with RSASSA-PSS signature key (id-RSASSA-PSS OID in
SubjectPublicKeyInfo) cannot be used because TLS defines no
corresponding signature algorithm.
As I didn't find any RFC's that counteract this, it seems impossible that TLS 1.2 uses PSS for authentication at all.
Now, on the other hand TLS 1.3 specifies that you should use PSS, which is not that surprising as TLS 1.3 very deliberately uses up to date cryptography at the cost of backwards compatibility.
For TLS 1.0 a very specific structure with both SHA-1 and MD5 is used. This standard predates SHA-2 and therefore it uses a more secure construct using two hashes that are separately pretty insecure. It also may use RSA encryption using PKCS#1 v1.5 padding as PKCS#1 v2.0 hadn't been introduced either. I don't see any changes for TLS 1.1 for RSA signing / encryption. Note that TLS 1.1 and 1.2 were mainly incremental updates of TLS 1.0 rather than the significant rewrite that TLS 1.3 is.
So basically, the use of RSA padding in TLS is directly tied to the standard itself. OpenSSL will try and follow that standard.
It might be that if you have a certificate that identifies the public key algorithm as PSS, see here. In that case an implementation may decide to go with that single choice, even though it is prohibited by the TLS 1.2 standard - I guess the correct response would be to throw an error, but may be a warning is generated instead.