# Decrypt digital signature using RSA public key with openssl

I have a digital signature that was created using the following algorithm: a SHA-256 hash of the body of the message is calculated. It is then signed using an RSA private key and the result is base64-encoded.

Now, I have the RSA public key corresponding to that private key which was used to encrypt the hash. I want to decrypt the digital signature using the RSA public key so that it gives me the SHA-256 hash of the body of message that was sent by the server. I can later compare this with the SHA-256 hash of the body of the message that was received.

I am unable to recover the SHA-256 hash (I'm expecting to get 64 hex characters). Here is what I am doing:

I save the public key in the following format in a file, pub.key:

-----BEGIN PUBLIC KEY-----
the key itself
-----END PUBLIC KEY------


With the following command:

openssl rsa -noout -text -pubin < pub.key


It tells me that the key is of length 2048 bits.

I save the base64-encoded digital signature in a file called sig.txt and then use the -verify option of openssl to retrieve the data.

openssl rsautil -verify -in sig.txt -inkey pub.key -pubin


This gives me the error:

RSA_EAY_PUBLIC_DECRYPT: data greater than mod len: rsa_eay.c


Please note that the signature in the file sig.txt is in base64 encoded form.

How can I use the RSA public key to decrypt this signature to recover the SHA-256 hash?

-
"It is then signed using an RSA private key" With which signature scheme? PSS? –  CodesInChaos Nov 18 '12 at 13:38
I do not know the signature scheme. The algorithm mentioned in the DKIM signature header field in email is: rsa-sha256 –  Neon Flash Nov 18 '12 at 14:30
You don't decrypt a signature, you verify it. –  Paŭlo Ebermann Nov 19 '12 at 14:34
Ok, so can you tell me how to verify it? The above command line is not working for me. –  Neon Flash Nov 20 '12 at 2:20
How many bits is the RSA key you are using? "data greater than mod len" usually indicates either a "toy" key that doesn't have enough bits or an attempt to operate on the actual data where you should be operating on a hash. –  David Schwartz Apr 1 '13 at 6:22

openssl rsautl expects a signature in binary format, not Base64-encoded. You should also check the signature scheme used. PKCS#1 v1.5 and PSS (PKCS#1 v2) are your best bets
The -verify switch is a bit misleading, the command only outputs the decrypted hash. You have to compare with the expected hash yourself.