What is the exact difference between the
CKM_RSA_X_509 mechanisms in the PKCS#11 v2.20 specification (Please explain with an example)?
As very clearly indicated by the specification,
CKM_RSA_X_509 performs "raw" RSA. Raw RSA is simply modular exponentiation. So it performs just the RSASP1 function in the PKCS#1 standards. This means that a user should - at the minimum - also provide a secure padding mechanism. Otherwise the conditions to perform a secure RSA signing operation are not met.
CKM_RSA_PKCS on the other hand also performs the padding as defined in the PKCS#1 standards. This padding is defined within EMSA-PKCS1-v1_5, steps 3, 4 and 5. This means that this mechanism should only accept messages that are 11 bytes shorter than the size of the modulus. To create a valid RSASSA-PKCS1-v1_5 signature, you need to perform steps 1 and 2 of EMSA-PKCS1-v1_5 yourself.
Both of these mechanisms are only building blocks for creating a signing algorithm. You won't get a valid PKCS#1 signature by supplying random data to either mechanism. These methods have been defined to give more control to the user of the mechanism.
Sometimes protocols such as older SSL specifications do use their own signature schemes (concatenating an MD5 and SHA-1 hash, leaving out the hash indicator). Those protocols cannot be supported without a direct implementation or lower level primitive being available.
Another reason that CKM_RSA_PKCS exist is that it may not be very efficient to perform the full PKCS#1 v1.5 signature generation on the security token. It is very inefficient to send an entire text document to a smart card. Just sending a hash structure calculated on the host computer is much faster.
A final note: you should never let an attacker be able to use
CKM_RSA_X_509 for the keys in your token, nor should you give him alter the padding you supply to that mechanism. In general it is better to use up to date mechanisms that represent complete signature schemes.