I am studying openssl 3.0 (Windows, C++) in order to encrypt data using RSA. More specifically, I need to sign a CertificateVerify server message in TLS 1.3.
The questions are:
1) how can I define padding mode (PSS) for RSA encryption for this aim ?
2) Should I to create EVP_PKEY_CTX* ctx or I need EVP_MD_CTX* mdctx only ? What's a difference between them ?
Code without padding definition is: `

// encryption (123)
    EVP_PKEY_CTX* ctx = EVP_PKEY_CTX_new(RsaPrivate, NULL);

// source data to encrypt
int SrcSize = 500;
unsigned char* SrcBuf = (unsigned char*)malloc(SrcSize);

/* Create the Message Digest Context */
EVP_MD_CTX* mdctx = EVP_MD_CTX_new();
assert(mdctx != nullptr);

/* Initialise the DigestSign operation with SHA-256 */
int res = EVP_DigestSignInit(mdctx, NULL, EVP_sha256(), NULL, RsaPrivate);
assert(res == 1);

/* Call update with the message */
res = EVP_DigestSignUpdate(mdctx, SrcBuf, SrcSize);
assert(res == 1);`

1 Answer 1


To clarify: "sign x" is usually shorthand for "generate a signature for [or on or over] the data x [with a stated or implied key]". You do not create a signature for (the data in) a CertVerify message, and couldn't because it would violate causality and the universe would be destroyed; rather 1.3 CertVerify contains a signature for a prefix plus the hash of the transcript so far. (In 1.2 and below, CertVerify is used only for client auth and is for the transcript hash; server signature is in the ServerKX message and is for the other data in the ServerKX, i.e. the server signs its ephemeral key and this is sometimes imprecisely described as signing the ServerKX message.)

Since OpenSSL implements SSL-now-TLS, and in fact that was its original purpose although it can be and now is used for other things as well, you can simply look at the libssl code to see how to do PSS (RSASSA-PSS) signature for this case.

Oh, and signature is NOT encryption -- not even "encryption with the private key" -- even though the OID names for v1.5 signatures say so, because they began to be assigned before this distinction was properly recognized. See my list at https://security.stackexchange.com/questions/159282/#159289 .

  • $\begingroup$ thank you for the link! As regards terminology - I meant that to create the sign we use encryption algorythm (RSA). I understand that we sign something (prefix+hash) inside CertificateVerify, but not this message at whole. May be I expressed myself incorrectly... If this terminology is incorrect - thank you for your clarification. $\endgroup$
    – LUN
    Commented Jun 17, 2023 at 11:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.