I am implementing RSA-OAEP with the mask generation function as specified in PKCS #1. It says after producing the variable T to "output the leading maskLen octets of T as the octet string mask". However, in one part of RSA-OAEP, where the mask generation function is used on a random seed, maskLen is greater than T. I don't believe I put in the wrong maskLen. RSA-OAEP does not work unless the output of the function is of the specified maskLen. Should I copy T to a byte array of length maskLen?
1 Answer
$\begingroup$
$\endgroup$
In MFG1, the size of $T$ in octets of the provisionally generated mask is never smaller than the specified $maskLen$ at the point where we want to
Output the leading $maskLen$ octets of $T$ as the octet string $mask$.
That follows from the fact that $T$ was built as the concatenation of $\lceil maskLen/hLen\rceil$ hashes, each of $hLen$ octets.
See the algorithm given in PKCS #1 V2.2, section B.2.1. Notice that the notation $\lceil maskLen/hLen\rceil$ means that the division is with the rounding upwards. In C and Java, if maskLen > 0, that is obtained as (maskLen - 1)/hLen + 1. This can not cause overflow, and the final + 1 nicely cancels with the prescribed - 1 in $\lceil maskLen/hLen\rceil-1$.
