Take the 2-minute tour ×
Cryptography Stack Exchange is a question and answer site for software developers, mathematicians and others interested in cryptography. It's 100% free, no registration required.

I have a digital signature that was created using the following algorithm: a SHA-256 hash of the body of the message is calculated. It is then signed using an RSA private key and the result is base64-encoded.

Now, I have the RSA public key corresponding to that private key which was used to encrypt the hash. I want to decrypt the digital signature using the RSA public key so that it gives me the SHA-256 hash of the body of message that was sent by the server. I can later compare this with the SHA-256 hash of the body of the message that was received.

I am unable to recover the SHA-256 hash (I'm expecting to get 64 hex characters). Here is what I am doing:

I save the public key in the following format in a file, pub.key:

-----BEGIN PUBLIC KEY-----
the key itself
-----END PUBLIC KEY------

With the following command:

openssl rsa -noout -text -pubin < pub.key

It tells me that the key is of length 2048 bits.

I save the base64-encoded digital signature in a file called sig.txt and then use the -verify option of openssl to retrieve the data.

openssl rsautil -verify -in sig.txt -inkey pub.key -pubin

This gives me the error:

RSA_EAY_PUBLIC_DECRYPT: data greater than mod len: rsa_eay.c

Please note that the signature in the file sig.txt is in base64 encoded form.

How can I use the RSA public key to decrypt this signature to recover the SHA-256 hash?

share|improve this question
    
"It is then signed using an RSA private key" With which signature scheme? PSS? –  CodesInChaos Nov 18 '12 at 13:38
    
I do not know the signature scheme. The algorithm mentioned in the DKIM signature header field in email is: rsa-sha256 –  Neon Flash Nov 18 '12 at 14:30
3  
You don't decrypt a signature, you verify it. –  Paŭlo Ebermann Nov 19 '12 at 14:34
    
Ok, so can you tell me how to verify it? The above command line is not working for me. –  Neon Flash Nov 20 '12 at 2:20
    
How many bits is the RSA key you are using? "data greater than mod len" usually indicates either a "toy" key that doesn't have enough bits or an attempt to operate on the actual data where you should be operating on a hash. –  David Schwartz Apr 1 '13 at 6:22
add comment

2 Answers

openssl rsautl expects a signature in binary format, not Base64-encoded. You should also check the signature scheme used. PKCS#1 v1.5 and PSS (PKCS#1 v2) are your best bets

The -verify switch is a bit misleading, the command only outputs the decrypted hash. You have to compare with the expected hash yourself.

Beware of the fact that the output will be the DER encoding of the SHA-256 hash.

share|improve this answer
add comment

I think you should decrypt it "cat sig.txt| base64 -d > sig1.txt" then try to verify it

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.