RSA Decryption using OpenSSL

Question

Trying to decrypt RSA Ciphertext with a private key using OpenSSl

Used rsatool to generate a .pem file for private key which is 1022 bit long.

Converted the original cyphertext 1023 bits long from just numbers to hex values of bytes

I have used the command: openssl pkeyutl -decrypt -in C:\Users\abc\Desktop\Cypher.txt -out C:\Users\abc\Desktop\result.txt -inkey C:\Users\abc\Desktop\key.pem

But received an error of: Public Key operation error 804C0000:error:0200006C:rsa routines:rsa_ossl_private_decrypt:data greater than mod len:crypto\rsa\rsa_ossl.c:406:

How can we solve this problem to get the plaintext?

Answer 1

Used rsatool to generate a .pem file for private key which is 1022 bit long.

Converted the original cyphertext 1023 bits long from just numbers to hex values of bytes

The result of an RSA operation is always a value smaller than the modulus size; that size is conventionally denoted as the key size.

If you do have an RSA key whose modulus is $< 2^{1022}$ (which is implied by the statement that the "private key [is] 1022 bit long), then it cannot generate a ciphertext $> 2^{1022}$ (which is implied by the statement that "the original cyphertext [is] 1023 bits long".

Because of this, OpenSSL is rejecting it, because this is impossible that this ciphertext correspond to this private key.

I suspect that you got either the private key or the ciphertext wrong (or the ciphertext doesn't, in fact, correspond to this key). I personally suspect that you might have gotten the key wrong - "1022 bits" is an unusual size"; 1024 bits is more common (if generally viewed as insecure nowadays).