4
$\begingroup$

I have a digital signature that was created using the following algorithm: a SHA-256 hash of the body of the message is calculated. It is then signed using an RSA private key and the result is base64-encoded.

Now, I have the RSA public key corresponding to that private key which was used to encrypt the hash. I want to decrypt the digital signature using the RSA public key so that it gives me the SHA-256 hash of the body of message that was sent by the server. I can later compare this with the SHA-256 hash of the body of the message that was received.

I am unable to recover the SHA-256 hash (I'm expecting to get 64 hex characters). Here is what I am doing:

I save the public key in the following format in a file, pub.key:

-----BEGIN PUBLIC KEY-----
the key itself
-----END PUBLIC KEY------

With the following command:

openssl rsa -noout -text -pubin < pub.key

It tells me that the key is of length 2048 bits.

I save the base64-encoded digital signature in a file called sig.txt and then use the -verify option of openssl to retrieve the data.

openssl rsautil -verify -in sig.txt -inkey pub.key -pubin

This gives me the error:

RSA_EAY_PUBLIC_DECRYPT: data greater than mod len: rsa_eay.c

Please note that the signature in the file sig.txt is in base64 encoded form.

How can I use the RSA public key to decrypt this signature to recover the SHA-256 hash?

$\endgroup$
7
  • $\begingroup$ "It is then signed using an RSA private key" With which signature scheme? PSS? $\endgroup$ Nov 18, 2012 at 13:38
  • $\begingroup$ I do not know the signature scheme. The algorithm mentioned in the DKIM signature header field in email is: rsa-sha256 $\endgroup$
    – Neon Flash
    Nov 18, 2012 at 14:30
  • 4
    $\begingroup$ You don't decrypt a signature, you verify it. $\endgroup$ Nov 19, 2012 at 14:34
  • $\begingroup$ Ok, so can you tell me how to verify it? The above command line is not working for me. $\endgroup$
    – Neon Flash
    Nov 20, 2012 at 2:20
  • $\begingroup$ How many bits is the RSA key you are using? "data greater than mod len" usually indicates either a "toy" key that doesn't have enough bits or an attempt to operate on the actual data where you should be operating on a hash. $\endgroup$ Apr 1, 2013 at 6:22

3 Answers 3

2
$\begingroup$

openssl rsautl expects a signature in binary format, not Base64-encoded. You should also check the signature scheme used. PKCS#1 v1.5 and PSS (PKCS#1 v2) are your best bets

The -verify switch is a bit misleading, the command only outputs the decrypted hash. You have to compare with the expected hash yourself.

Beware of the fact that the output will be the DER encoding of the SHA-256 hash.

$\endgroup$
1
  • $\begingroup$ rsautl -sign/verify doesn't support PSS, although pkeyutl does (and can either 'recover'=show the hash or check it, for v1.5). For v1.5 recovery gives DER of ASN.1 DigestInfo containing hash OID and hash value; for PSS recovery is not supported, but PSS padding does only hash not DigestInfo. PSS is technically v2.1 up, not v2.0 (which was for OAEP encryption). $\endgroup$ Oct 4, 2019 at 1:38
1
$\begingroup$

Like @Tiger and @Aym_C said, first you need to convert the base64 into binary. Then I recommend feeding it to Openssl's rsautl with the -raw flag set.

Here's a one-liner that should accomplish the task:

base64 -d sig.txt | openssl rsautl -verify -inkey pub.key -pubin -raw -hexdump
$\endgroup$
0
$\begingroup$

I think you should decrypt it "cat sig.txt| base64 -d > sig1.txt" then try to verify it

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.