I have implemented signature scheme 2 from ISO/IEC 9796-2 in C using OpenSSL for the underlying crypto operations.
To generate the message representative (
F in the standard), I just call
PKCS1_MGF1() (defined here), and then XOR that to the data to be signed, excluding the hash and the trailer, as the standard says.
Then, I get that blob and run it through
RSA_private_encrypt(), setting the padding flag to
RSA_NO_PADDING (docs here).
It turns out that, for some inputs, the resulting message representative is bigger than the modulus when converted to a number, which causes
RSA_private_encrypt() to fail.
NOTE: the message representative is computed according to the standard, it is always 128 bytes in length (just like the modulus), and it matches the samples provided in Appendix E, so I presume there are no bugs in the way I generate
My key is a 1024-bit RSA private key generated with
openssl genrsa, with
e = 65537, which is an odd number. To the best of my understanding, given that
e is odd (which the standard calls
v, but they're otherwise the same thing) I can use the "alternative signature production function" (defined in Appendix B.6), which is, well, basically textbook RSA.
Is there anything I have misunderstood that is causing this issue?
If my implementation is correct, then how could I deal with these
Fs that are bigger than the modulus?