I'm trying to verify a RSA signed message, however, the API that I have access to in that environment only has encrypt and decrypt. I think I understand that verification and encryption with public key are similar. Actually, running the following seems to work:

openssl rsautl -sign -in clear -inkey private_key.pem -out signed
openssl rsautl -encrypt -pubin -inkey public_key.pem -in signed -raw -hexdump

This second command will get the initial clear, padded (then I just need to unpad and verify that it matches the clear). However, the -raw is where I encounter an issue. As the signed message is the size of the key, I can't encrypt it (the API will refuse, as does openssl if you omit -raw).

I'm not sure on how to proceed. I feel that what I'm trying to do should be possible.

  • $\begingroup$ FYI rsautl with default pkcs1 padding does not implement all of SSA-v1_5, specifically not en/decoding the (supposed) hash with the AlgId. Thus the signature it generates won't verify on a system that follows the standard, and it doesn't really simulate signing on a system that follows the standard. See superuser.com/questions/943972/… . However, as @Maarten answered, your attempt fails before even reaching this problem. $\endgroup$ – dave_thompson_085 Sep 3 '15 at 9:38

The API you are trying to use automatically pads (probably RSAES-PKCS1-v1_5 padding). Instead you require it to verify and remove the padding during signature verification.

If you cannot skip the padding then you cannot use the API. Try and find a library that actually can verify instead. You shouldn't program RSA itself, but in the end it's just an algorithm. As long as you have access to the key - and preferably a big integer math library - you should be able to perform RSA modular exponentiation and verification / removal of the padding yourself.

You cannot just magic away 11 bytes that are added to the plaintext before modular exponentiation. OAEP would pad even more.

You could try and decrypt the signature with the public key and hope that the API detects the public key and performs the verification and unpadding instead itself, but that's a long shot. Note that you should get an error message if the padding cannot be verified.

Note that Diffie-Hellman also performs modular exponentiation. Sometimes you can use DH functionality to "raw decrypt" an RSA signature.

  • $\begingroup$ To be sure I understand: normal encrypt=pad then encrypt and verify=encrypt then unpad? $\endgroup$ – Agnar Sep 3 '15 at 7:29
  • $\begingroup$ Kind of yes. "encrypt = pad then modular exponentiation" for encrypt. And afterwards the I2OSP method is used to make the ciphertext as long as the modulus in bytes. Obviously the verification should check the padding and the hash value as well. The padding methods for signature generation and encryption differ though. See RFC 3447 $\endgroup$ – Maarten Bodewes Sep 3 '15 at 7:52
  • $\begingroup$ What I ended up doing thanks to that answer: I had access to the code of the API, so I copied the code used for encryption, removed the check on the size, removed the padding at the beginning and added a unpad at the end. As I didn't touch any crypto calls, I believe I haven't compromised the security of the operations. $\endgroup$ – Agnar Sep 8 '15 at 22:43
  • $\begingroup$ For verification this is probably OK. Beware that for signing that you probably be better off using the decryption code as RSA private key operation are vulnerable to e.g. timing attacks. Glad you got it solved. $\endgroup$ – Maarten Bodewes Sep 9 '15 at 0:15
  • 1
    $\begingroup$ @Agnar: there are many ways your implementation can be wrong; in particular, you should not be removing the padding, you should be checking it! This extremely classical mistake it known to allow for forgery!! There are many other classical goofs (including allowing $s+N$ as a signature where $s$ is the genuine signature, which breaks security in some definition of that). $\endgroup$ – fgrieu Oct 14 '15 at 15:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.