Curious behavior of EVP_DigestSign for DKIM

Question

Difficulties porting code to OpenSSL 3 may raise some doubts on digest signing. Before version 3 came, digest signing was the only way to use ED25519. On version 3, RSA_sign() being deprecated, I tried to re-use ED25519 code with RSA keys. The code is like so:

    char *digest;              // digest already computed from
    size_t diglen;             // normalized message header

    EVP_PKEY *pkey;            // given

    size_t crypto_outlen;      // expected
    unsigned char *crypto_out; // pre-allocated EVP_PKEY_size(pkey);

    EVP_MD_CTX *md_ctx = EVP_MD_CTX_new();
    if (md_ctx == NULL) goto error;

    int status;
#if 0
    if (signalg != ED25519SHA256)
        status = EVP_DigestSignInit_ex(md_ctx, NULL,
            signalg == SHA1? "SHA1": "SHA256",
             NULL, NULL, pkey, NULL);
    else
#endif
        status = EVP_DigestSignInit(md_ctx, NULL, NULL, NULL, pkey);

    if (status == 1)
        status = EVP_DigestSign(md_ctx,
            crypto_out, &crypto_outlen,
            digest, diglen);
    EVP_MD_CTX_free(md_ctx);
    if (status != 1) goto error;

The code continues to work with ED25519 keys. However, with RSA keys it produces results different from the test suite values. I tried enabling the extended initialization version, which sets the type of SHA, and the results differ from both the prior results and the correct values in the tests.

DKIM has a complicated way of producing the hash, which provides for simple or relaxed canonicalization —how to compress white space, lower case and similar nuances. Originally, only RSA signatures were allowed, with either SHA1 (later deprecated) or SHA256. In 2018, RFC 8463 introduced ED25519, but it preserved the known SHA256 hashing algorithm. I had the impression that such usage looses some of the collision resistance properties of Schnorr signatures. I'm no cryptography expert, but may appreciate a short note on this, especially if it sheds some light on the questions that follows.

In my mind, I conceive signing like a sort of multiplication of the digest by the key. Accordingly, the signing function should only need to know the digest length. And, indeed, with ED25519 it works without knowing how the digest was obtained.

Is there any hope to do RSA and ED25519 signatures using (almost) the same signing code? Perhaps I could try tweaking the EVP_MD_CTX object. Or I could turn to EVP_PKEY_sign(), but in that case should I do so for RSA only, or would ED25519 still work with that function? Hm...