The first thing to clarify is the definition of “EdDSA”. EdDSA was introduced in Bernstein et al.'s High-speed high-security signatures in 2011. Various parameters it implicitly assumed were declared more generally in the 2015 paper Bernstein et al., EdDSA for more curves. This culminated in an RFC Edwards-Curve Digital Signature Algorithm (EdDSA), RFC 8032 from 2017.
All of them describe EdDSA abstractly and then instantiate it with a set of concrete parameters. Thus, Ed25519 and Ed448 are specific sets of parameters for EdDSA that form a concrete signature scheme that can be implemented.
- Is it possible to use EdDSA with a custom algorithm (signature scheme) for instance a different curve and a different hashing algorithm like SHA-1?
Yes, this is possible, assuming you do not mean the specific instantiations in the first paper and RFC 8032.
In fact, the Monocypher cryptography library's crypto_sign() does exactly that, swapping out SHA-512 in favor of BLAKE2b.
- Is this not possible because of an RFC standard?
As noted above, this is possible, but the result obviously wouldn't be able to be called “Ed25519” or “Ed448” because it'd be incompatible with those specific instantiations.
- Does this mean that EdDSA is rigid by design?
It isn't, no. The 2015 paper even goes out of its way to add a bunch of.
EdDSA as specified in all of these documents is somewhat inflexible with regards to the underlying curve, however: It assumes that the cofactor $h=2^c$ has $c\in\{2,3\}$, which rules out prime-order Weierstrass curves for example. The required modifications to be able to not require $c$ with a prime-order curve are both trivial and obvious, but this specific set of permissible values for $c$ mandated by all specifications of EdDSA.
- Does this have to do with OpenSSL not having the feature implemented?
No, nobody specifying EdDSA cared what OpenSSL does because that'd be putting the cart before the horse: OpenSSL can't implement EdDSA before it is specified.
