What is the difference between openssl functions -
d2i_PUBKEY and what scenarios decide which should be used?
d2i_PublicKey was being used in the application which was returning a valid
EVP_PKEY object until a specific testcase was found wherein it returned NULL. If the function call is changed to
d2i_PUBKEY it returns a valid
EVP_PKEY. But it is specific to this one testcase only and the rest of them work with the other function -
d2i_PublicKey. It seems that some initial checking/processing needs to be done on the arguments before deciding which function to call. Has anyone faced similar issue before and could help with the relevant information or code?
Variant 1: EVP_PKEY_result_obj = d2i_PublicKey(keyType /*EVP_PKEY_RSA*/, nullptr, &publicKeyData, publicKey.size()); Variant 2: EVP_PKEY_result_obj = d2i_PUBKEY(nullptr, &publicKeyData, publicKey.size());
The 1.1.0 branch documentation   states the following regarding these functions:
d2i_PrivateKey() decodes a private key using algorithm type. It attempts to use any key specific format or PKCS#8 unencrypted PrivateKeyInfo format. The type parameter should be a public key algorithm constant such as EVP_PKEY_RSA. An error occurs if the decoded key does not match type. d2i_PublicKey() does the same for public keys.
i2d_PUBKEY() decode and encode an
EVP_PKEY structure using SubjectPublicKeyInfo format. They otherwise follow the conventions of other ASN.1 functions such as
It isn't clear how to interpret the above information to help understand the ongoing issue.
If there is any other information needed to further understand the issue, please let me know - I'll add it here.