OpenSSL RSA same plaintext but different ciphertext

Question

how OpenSSL RSA work ?

I generate public (n,e) and private key (n,d) then I encypted a file by :

openssl rsautl -encrypt -in plaintextFile -inkey privkey.pem -out cipher00

let's note the result C

and I tried to decrypt it by doing C^d (modulo n) but it doesn't work.

I think this is because OpenSSL adds some random value to my plaintext before the encryption. If I run the previous command once again on the same plaintext I have a different ciphertext.

How can I remove this random during the decryption?

Answer 1

The "normal", unmodified RSA (called textbook RSA) is susceptible to some attacks. We need to change it slightly to avoid this problems. The question Definition of Textbook RSA and the Wikipedia lists some possible attacks. In practice a special padding algorithm is used, like the Optimal asymmetric encryption padding (OAEP). The documentation of the RSA_public_encrypt function in OpenSSL shows the possible padding algorithm which can be used.

So yes, OpenSSL adds some random information to the plaintext. Without this, RSA would be deterministic: The same ciphertext would reveal that the plaintext was also identical. If you want to decrypt the ciphertext, then you have to either use OpenSSL again, or remove the padding manually.