I need MGF1 and OS2IP functions, as defined in the PKCS #1 v2.1: RSA Cryptography Standard.
Is there anything available in OpenSSL, or any other open C library?
You can find MGF1 in OpenSSL. Source file "crypto/rsa/rsa_oaep.c" has a function called "PKCS1_MGF1" that implements MGF1. The function signature is:
See the wrapper function "MGF1" defined right below it for a more general case of using it.
As for OS2IP, I don't see an explicit method for converting to/from it in OpenSSL, presumably because the integers start and end in that format, so they never convert them.
OS2IP and I2OSP are a pair of straight-forward methods of encoding and decoding a large number to an octet string. The idea for OS2IP is that you simply represent the integer in big endian byte order. How you would use OS2IP to encode an integer depends entirely on how the integer is original represented before conversion.
The bottom line for OS2IP/I2OSP is that it's more a function of your original integer than a function of RSA. Find a way to convert the original integer to/from a big endian byte string, either by using a "toBigEndian" / "fromBigEndian" type function for your original integer or by making one manually. I don't think you'll find a generic method anywhere.
To do it manually: If you understand how the original integer format works, you can manually convert bytes to/from the original format and the octet string. If you can do multiplication, division, addition, and subtraction on the original integer, you can just calculate the octet stream manually with a few lines of code and a loop using the OS2IP description.