I have implemented signature scheme 2 from ISO/IEC 9796-2 in C using OpenSSL for the underlying crypto operations.

To generate the message representative (F in the standard), I just call PKCS1_MGF1() (defined here), and then XOR that to the data to be signed, excluding the hash and the trailer, as the standard says.

Then, I get that blob and run it through RSA_private_encrypt(), setting the padding flag to RSA_NO_PADDING (docs here).

It turns out that, for some inputs, the resulting message representative is bigger than the modulus when converted to a number, which causes RSA_private_encrypt() to fail.

NOTE: the message representative is computed according to the standard, it is always 128 bytes in length (just like the modulus), and it matches the samples provided in Appendix E, so I presume there are no bugs in the way I generate F.

My key is a 1024-bit RSA private key generated with openssl genrsa, with e = 65537, which is an odd number. To the best of my understanding, given that e is odd (which the standard calls v, but they're otherwise the same thing) I can use the "alternative signature production function" (defined in Appendix B.6), which is, well, basically textbook RSA.

Is there anything I have misunderstood that is causing this issue?

If my implementation is correct, then how could I deal with these Fs that are bigger than the modulus?

  • $\begingroup$ It would be nice if you reproduced the examples as ISO/IEC 9796-2 is payware. Note that there is a difference between the N the number and |N|, the size in bits. Basically, independent on N, you can encrypt (perform modular exponentiation) on any 1023 bits, not 1024 bits. This is because some 1024 bit numbers that are lower than N (when seen as a unsigned big endian number) can be encrypted otherwise. And N depends on the key pair generation - the choices of $p$ and $q$. $\endgroup$
    – Maarten Bodewes
    Commented Oct 29, 2019 at 14:11

1 Answer 1


ISO/IEC 9796-2 is a digital signature scheme giving message recovery. That is: able to embed some of the message in the signature, thus lowering communication or/and storage requirements. There are a lot of options:

  • Total recovery is when the message is fully embedded in the fixed-size signature; this limits the message size to a certain capacity. This opposes to Partial recovery were the message size exceeds the capacity: the recoverable part $M_1$ of the message (up to capacity) is embedded in the signature, and the (non-empty) non-recoverable part $M_2$ must somewhat be made available to the verifier independently from the signature (often: along).
  • There are 3 schemes:
    • #1 is the original, simplest, but is at least theoretically broken in a chosen-messages setup. It is used in in EMV, Passport, and Digital Tachograph.
    • #2 is inspired by RSASSA-PSS, and uses random salt and MGF1.
    • #3 is simply #2 with no salt.
  • The hash used for signature. A common choice for 1024-bit modulus would have been SHA-1, but both are obsolete. A modern choice is SHA-256.
  • If that hash is assumed known by the parties (Trailer field option 1) or encoded in the message representative trailer (rare: the motivation for ISO/IEC 9796-2 is compact signed messages, so the overhead of explicitly encoding the hash is unwelcome, plus that's not a good idea from security and simplicity standpoints).
  • For schemes #2 and #3: which hash is used for MFG1. Two popular choices are SHA-1, and the same hash as in as the signature.
  • Salt size for scheme #2. The more salt, the simpler/stronger the security argument, but the lesser the capacity.
  • Odd or even public verification exponent $v$, that is RSA or Rabin. $v=2$ provides the fastest standardized signature verification around.
  • Use of "Alternative" signature (RSASP1 and RSAVP1 in PKCS#1, basically big-endian textbook RSA with left-padding with zeroes up to a fixed size), or non-alternative which saves one bit in the signature (I have only seen that with even public exponent, and in test vectors).
  • Modulus size: many implementations restrict to modulus a multiple of 8 bits, or a larger power of two.
  • Message alignment: the standard supports any size, but many implementations restrict to bytes.

When using byte-aligned message, $k$-bit modulus (multiple of 8), $L_h$-bit hash (multiple of 8), salt size $L_s$ (multiple of 8, possibly 0), Trailer field option 1, the capacity is $k-L_h-L_s-16$ bits.

There are variations in usage:

  • How the data to be signed is split in recoverable and non-recoverable parts of the message: the standard defines that the recoverable part of the message is at the beginning, but sometime it is desirable that the start of the payload data is readable before signature verification (e.g. because it includes a key index), therefore the recoverable part of the message sometime is gathered from the end of the payload data.
  • Signature before or after the non-recoverable message in signed data.
  • Tolerating use of partial message recovery mode when the message size is exactly the capacity.
  • Perhaps, tolerating a recoverable message $M_1$ shorter than the capacity including with a non-empty non-recoverable message $M_2$.

It turns out that, for some inputs, the resulting message representative is bigger than the modulus when converted to a number.

That's a clear indication that the message representative is not computed correctly.

To generate the message representative ($F$ in the standard), I just call PKCS1_MGF1() and then XOR that to the data to be signed, excluding the hash and the trailer, as the standard says.

Based on that oversimplified description, the problem likely is that the output of MFG1 is considered to be bytes, when the standard wants it one bit less than a multiple of 8 (when everything is byte-aligned). You want to zero the first/left bit of what MFG1 produces before the XOR, or equivalently zero the high-order bit of the message representative after the XOR. That won't lose message data, since that bit is part (or on the left of) the header.

I can use the "alternative signature production function"

Yes. That's by far the most common when public exponent is odd.

  • $\begingroup$ Thanks. I looked closer and the message representative was indeed not being computed correctly. I skipped a detail which I thought, was minor. I need to be careful with the information I post here as ISO/IEC 9796 is subject to royalties, but for those who might be struggling with this or a similar problem in the future, everything is in the standard. You just need to follow it closely, to the detail. If you follow all the steps, the problem I describe can't happen. $\endgroup$
    – aja
    Commented Oct 30, 2019 at 8:07
  • $\begingroup$ To clarify "ISO/IEC 9796 is subject to royalties": the documents forming ISO/IEC 9796, including ISO/IEC 9796-2, are available for a fee, and redistribution of these documents is forbidden or subject to royalties. But use of all the techniques mentioned in ISO/IEC 9796-2:2010 and earlier is free (I'm not sure for ISO/IEC 9796-3). Short citations of standards for the purpose of discussion are OK. Also: ISO/IEC 9796-2:2002 is floating on the internet; changes since that include incorporation of the ASN.1 grammar defined by ISO/IEC 9796-2 Amd1:2008 and are otherwise editorial AFAIK. $\endgroup$
    – fgrieu
    Commented Oct 30, 2019 at 12:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.