Why is data signed with SHA256-RSA-PKCS and digest signed with RSA-PKCS different?

Question

I need to sign some data using RSA. I have the digest that I can sign but the signature turns out different from the signing source data.

There is data in data.bin and its sha256 digest in digest.bin

$ xxd -p -c32 digest.bin
19d512fc649e1668eb84741284ad95ec03f3225719a83e40389dbade4eabe5ed
$ sha256sum data.bin 
19d512fc649e1668eb84741284ad95ec03f3225719a83e40389dbade4eabe5ed  data.bin

Why is the signature different when signing data using SHA256-RSA-PKCS and when signing a digest with RSA-PKCS? In both cases, the same sha256 digest should be created and then signed.

$ pkcs11-tool --module /usr/lib/libIDPrimePKCS11.so --pin=$PIN --slot 0x11 --sign --input-file data.bin --mechanism SHA256-RSA-PKCS |base64 -w0
Using signature algorithm SHA256-RSA-PKCS
N4XkgBhyu5fAFTSraEj4RlKeyKy2fyUD4kv9QgBVmRl/u0jDYuPdQ/Gr0Hf2wRD8B3AWe98Y3WWtv5/c2NILosG5VzqrQEKVFKLCLkdpV7cQoSDkzAv0YjRRWKMJPr1Y9Vwi15yua17W8JqnqAc0D/Mda68qgugxgCzfwC+BkWnjsN3EaxixIzwZ9aMbfuBbA8R/+uRZMdPHc6w/aFIhSM9sS3jyeb2ZYy7semPlFMAx1BQx4C2uGlEW5GI7BCq1sTCchuB0baR4WX+yoHp+bRsJHWjEcXmgFloxfb6k0n3rvZOZboO0h9UPbxlxFxHou1v6UqymNgnt+oC1KkclRQ==

$ pkcs11-tool --module /usr/lib/libIDPrimePKCS11.so --pin=$PIN --slot 0x11 --sign --input-file digest.bin --mechanism RSA-PKCS |base64 -w0
Using signature algorithm RSA-PKCS
RseZYWjkyNRcU1lI1KR+I+0C1jC+FVp2kTAhyOUsE2x6UGf+UMSf3kqD4KGBb+eIryXDFTzNsdo8XroUtWvYQvHgPLPaRQ8xfPtTOZS/crQY168S2lk4buAeDZWat/xiGeXgU4rL7hdbjojdkfbeGWHfIERcese6wv521t9x9w5K3XGz50Ec9Y/C4cPPo4xuQMSQlCrx1iIlfeGbkElxGIdIkZvt2nTrhl/ico5YCbTJfuDqZaaaJxPkXcfgAcvFVU/kILi7U3XQCvz7zohZoRj4vkeTutanW6I7qGyWvIcxXGJx29MInF3pohT0+KUzegN4A1GfkW5hfk5HyCP2Gw==

public key is

-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwzz8XZ5iDjWlLZOfXgzh
VsyFcx85QEhgrcJJXkFV2k+vmVUA1R4oZPuvE+cF13ymh05+GLM0OHHzrohSrbv9
3AUmkWLPY04jF0NXSEAd180ImJ84Y2lSaqo4op+l3cPjIWeeBN0DPX78GdLGJnIr
Fq51TwKcdtjgQMvfWDuxZbjo2C20QQIIzp0cWGTsWOBFamjOJGhfRcQ/TT4MDA2h
//813N2HqXnlAtEgvlLH1Jv+VNROd/WIN+CMo0NLDQGS0AhVZPctatNqH5mXwklZ
goyv62WY284EIZLC1nTgMWxdQQIR6yRyJmQwp7zLs/Fk9rYZDHEOTuILEHd/7dPB
7wIDAQAB
-----END PUBLIC KEY-----

Answer 1

TLDR: it's the DigestInfo

The hash-and-RSA-PKCS schemes defined in Current Mechanisms 2.1.14 do the full scheme of RSASSA-PKCS1-v1_5 which is now defined in RFC8017 primarily in 9.2 as a combination of EMSA-PCKS1-v1_5 'encoding' (usually called padding) plus the RSASP1 and RSAVP1 primitives. (In prior versions the document was organized differently, but the substance was the same.) Concretely this consists of:

#1 hash the data

#2 encode the hash value, plus the hash algorithm, in an ASN.1 structure DigestInfo in DER; since each hash algorithm has constant output size, this amounts to prepending a fixed prefix depending only on the algorithm, as given in note 1

#3-5 prepend padding of the form 00 01 FF...(at least 8) 00

(8.2.1#2) treating the result as a number m, apply RSASP1 which does m ^ d mod n (or for verify 8.2.2#2 apply RSAVP1 which does s ^ e mod n; this is stated as before the three padding steps above but actually can just as well be after)

OTOTH RSA-PKCS defined in 2.1.6 for Sign/Verify does ONLY the latter two of those steps, not the first two. As the text says: "This mechanism corresponds only to the part of PKCS #1 v1.5 that involves RSA; it does not compute a message digest or a DigestInfo encoding as specified ... in PKCS #1 v1.5 ." You externally hashed the data which is the first step, but did not do the DigestInfo which is the second (required) step, and thus got the wrong result.

Except for being PKCS11 (hardware) rather than software, this is crossdupe at least:
https://stackoverflow.com/questions/521101/using-sha1-and-rsa-with-java-security-signature-vs-messagedigest-and-cipher
https://stackoverflow.com/questions/9951559/difference-between-openssl-rsautl-and-dgst
https://stackoverflow.com/questions/38767660/multiple-openssl-rsa-signing-methods-produce-different-results
https://stackoverflow.com/questions/63582544/how-does-jwt-implement-rsa256-signature-verification-in-nodejs
https://stackoverflow.com/questions/62955968/how-can-realize-openssl-pkeyutl-sign-by-java
https://stackoverflow.com/questions/48013643/get-the-sha1-hashed-value-from-xml-signature-value
https://stackoverflow.com/questions/43490188/separate-digest-signing-using-java-security-provider
https://stackoverflow.com/questions/57929493/openssl-digitally-sign-digest-only
https://stackoverflow.com/questions/9380856/openssl-signature-difference-when-using-c-routines-and-openssl
https://stackoverflow.com/questions/13419201/why-are-the-rsa-sha256-signatures-i-generate-with-openssl-and-java-different
https://stackoverflow.com/questions/15092298/signing-20-byte-message-with-256-bit-rsa-key-working-with-openssl-exe-but-not-in
https://superuser.com/questions/943972/what-is-the-difference-between-openssl-pkeyutl-sign-and-openssl-rsautl-sign