0
$\begingroup$

In this document, it shows the EM will have one zero octet padding prefixed.

                     +----------+---------+-------+
                DB = |  lHash   |    PS   |   M   |
                     +----------+---------+-------+
                                    |
          +----------+              V
          |   seed   |--> MGF ---> xor
          +----------+              |
                |                   |
       +--+     V                   |
       |00|    xor <----- MGF <-----|
       +--+     |                   |
         |      |                   |
         V      V                   V
       +--+----------+----------------------------+
 EM =  |00|maskedSeed|          maskedDB          |
       +--+----------+----------------------------+

However, in this document, page 37, it does not have this 00 octet.

For the standard (or the implementation of openssl) RSA PKCS#1 OAEP padding, which one (with 00 octet or without octet) is the standard/correct?

In fact, I've done the following tests:

  1. I use RSA_public_encrypt(RSA_PKCS1_OAEP_PADDING) and find out the max message length is 214, which should be yielded by 256-20*2-2(k - 2hLen - 2).However, per https://www.openssl.org/docs/man1.0.2/man3/RSA_private_decrypt.html, it mentions "EME-OAEP as defined in PKCS #1 v2.0" so I think it should be based on rfc2437. But the max length for rfc2437 is "emLen-1-2hLen", which is a conflict?

  2. I use RSA_public_encrypt(RSA_PKCS1_OAEP_PADDING) to encrypt a message(8 bytes only) and then use RSA_public_decrypt(RSA_NO_PADDING) to decrypt it to see the output. I notice there always be a leading 00 octet padded. This suggests it is based on rfc3447(v2.1). I also have no idea about this.

Improve this question
$\endgroup$
1
  • $\begingroup$ It matters what version of PKCS1 you use because 2.0 used one notation and 2.1 and 2.2 used a different but functionally equivalent notation. In 2.0 (rfc2437) EME-OAEP-ENCODE is invoked with emLen = k-1 where k is the size of the modulus n=pq, and the 'message' is limited to emLen-2hLen-1 = k-2hLen-2 . EM is then k-1 octets, but when converted by OS2IP in step 2 of 7.1.1 it results in the 'k-size' integer equal to 00 || EM, and on decryption step 4 of 7.1.2 enforces this. Compare crypto.stackexchange.com/questions/40032/… . $\endgroup$
    – dave_thompson_085
    Dec 31, 2020 at 23:24

1 Answer 1

Reset to default
1
$\begingroup$

According to RFC 2437...

  • RSAES-OAEP can operate on messages of length up to $k-2-2\cdot hLen$ octets, where $hLen$ is the length of the hash function output for EME-OAEP and $k$ is the length in octets of the recipient's RSA modulus.
  • The padding string $\text{PS}$ in EME-PKCS1-v1_5 is at least eight octets long, which is a security condition for public-key operations that prevents an attacker from recovering data by trying all possible encryption blocks.
    1. Generate an octet string PS consisting of $emLen-||M||-2\cdot hLen-1$ zero octets. The length of $\text{PS}$ may be $0$.
  • concisely, the length of zero padding will be $${emLen-||M||-2\cdot hLen-1}$$

Running with OpenSSL;

openssl rsautl -decrypt -inkey priv2.txt -in cipher.txt -raw -hexdump

and the first line of the output is;

0000 - 00 25 2b 2c e4 b5 a8 6c-a1 d1 cc bd 0b 26 d2 9c .%+,...l.....&..

You can see that the first byte above is 0x00

Improve this answer
$\endgroup$
8
  • 1
    $\begingroup$ Quick question, why not reference the newer RFC 8017? $\endgroup$
    – DannyNiu
    Dec 31, 2020 at 6:18
  • $\begingroup$ Thanks SSA and DannyNiu. My confusion comes from the following: Per openssl.org/docs/man1.0.2/man3/RSA_private_decrypt.html, for the OAEP, it states it is "EME-OAEP as defined in PKCS #1 v2.0 with SHA-1,...". Therefore, it should be based on RFC 2437. However, when I RSA_private_encrypt(RSA_PKCS1_OAEP_PADDING) and then RSA_private_decrypt(RSA_NO_PADDING), I always see a leading 00 octet is padded. Any idea? $\endgroup$
    – CHL
    Dec 31, 2020 at 7:33
  • $\begingroup$ @CHL, Are you getting 00 with any arbitrary plaintext? Could you provide us your sample plaintext and openssl command you use for encryption and decryption? $\endgroup$
    – SSA
    Dec 31, 2020 at 10:13
  • $\begingroup$ you will get 00 always padded at the beginning of padded string. Is that you are asking? $\endgroup$
    – SSA
    Dec 31, 2020 at 10:47
  • $\begingroup$ Yes. I always get 00 when I use RSA_private_decrypt(RSA_NO_PADDING). $\endgroup$
    – CHL
    Dec 31, 2020 at 11:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.