I've been searching for some time and have found no way to create a working certificate and private key pair with a DH or ECDH public key using OpenSSL. Once I do this, I'd like to sign it with an RSA key, too. Does OpenSSL support this? If so, how can I do it?
DH: OpenSSL commandline has three options for creating certs, but all of them either selfsign the cert or require a selfsigned CSR, and DH can't do either of those. OpenSSL library called from a program you write can construct an
X509 object (cert) containing a DH publickey, subject and other attributes as you specify, signed by an RSA key corresponding to a parent (CA) cert. Look at the code for
x509_certify as a (very approximate) starting point. Remember that if the same parent key/cert was or will be used to issue other certs as well you should keep all serialnumbers unique, perhaps using the same simple file
(update) In 1.0.2 this changed;
x509 -req -CA[key] now has an option
-force_pubkey which allows you to create a CSR with a dummy key that supports signing but issue the cert for the DH key. (I didn't notice right away because the usage message wasn't updated until 1.1.0.)
ECDH: the same key format is used for ECDH and ECDSA and the cert is distinguished if at all only by whether KeyUsage extension permits keyAgree versus digSign. Therefore with commandline you can generate (in order) EC parameters and an EC keypair, an ECDSA-selfsigned CSR, and a cert signed by a parent key&cert either RSA as you asked or ECDSA, optionally containing a KeyUsage extension as you desire (and other extensions also). The two options and variations are the same as for entity RSA except using an EC keypair -- which is generated using EC parameters (aka curve) where RSA keypair is generated using only a bitsize (which is sometimes defaulted).
protocols: If you intend to use these certs for SSL/TLS, note that OpenSSL does not yet implement the static-DH ciphersuites, even though it has long implemented the primitives needed by those suites -- and DHE and DHanon which are implemented. Apparently for the last 15 years there has been essentially no demand; this suggests that the peer systems you can communicate with using static-DH is probably the empty set. The probably-soon next release 1.0.2 is planned to add them. (update: it did, in Jan. 2015) OTOH static-ECDH has been implemented since EC ciphersuites as a whole were implemented; the code was actually in 0.9.8 but disabled by default, and since 1.0.0 it has been enabled by default except that RedHat packages before late 2013 removed (all) EC due apparently to patent concerns. (update) However 1.1.0 released in Aug. 2016 removed both static-DH and static-ECDH ciphersuites.
curve format: for EC (including ECDH) it is possible to represent some standard and popular EC curves either as a OID that maps to a code set by rfc4492, or as 'explicit' parameters (underlying field prime or polynomial, equation coefficients, base point coordinates, order and cofactor). These produce the same EC computation but are NOT equivalent in SSL/TLS protocol. In particular in some cases libssl will successfully use a cert/key with the 'named' form curve (if agreed by the peer system, of course) but not one with the 'explicit' form of the same curve. Use a standard curve (i.e. don't generate your own) and be sure not to convert the parameters to explicit form.
EDIT 2017-10: belatedly found crossdupe https://security.stackexchange.com/questions/44251/openssl-generate-different-types-of-self-signed-certificate
Edit: Additional answer to question(s) in comment by Martin Del Vecchio which initially appeared similarly but is in fact completely different.
This new problem is not DH or even ECDH, it is ECDHE. I'm not sure exactly what you are doing so I'll just start from the beginning and work through.
DH and ECDH are different algorithms. They are based on similar mathematical concepts, but they use completely different data. Classical or "integer" Diffie-Hellman uses a usually-smaller subgroup of the multiplicative group of integers modulo a large prime. Elliptic-curve Diffie-Hellman uses a usually-maximal subgroup of the points on an elliptic curve over an underlying field. A DH key or cert cannot be used for ECDH, and an ECDH key or cert cannot be used for DH.
ECDH and ECDHE are different protocols. SSL/TLS has three main key-exchange mechanisms that use the ECDH algorithm: "fixed" or "static" ECDH, represented by just ECDH in cipher suite names; "ephemeral" ECDH represented by ECDHE; and "anonymous" ECDH represented by ECDH_anon in the standard (RFC) names but AECDH in OpenSSL. The first two have variants depending on what authentication is used; anonymous does no authentication, which in practical use on the internet isn't secure, and thus is disabled by nearly all implementations including OpenSSL by default.
What Firefox offers: I don't use WebRTC, but for https my Firefox 39.0 offers 11 suites: 6 ECDHE (3 ECDHE_ECDSA and 3 ECDHE_RSA, almost as you had in your comment except the sixth one should be c012 not a repeat of c013), 2 DHE_RSA suites (that's integer-DH not ECDH), and 3 plain RSA suites (not using any kind of DH). I find it surprising NSS (which is what FF uses for SSL/TLS) would be different for DTLS than TLS; perhaps it's because DTLS started at the same time and protocol level as TLS 1.1, which is well correlated with ECC although it doesn't technically require it. But even if ECDHE isn't required it's still preferable both in itself (PFS and efficient) and because FF/NSS offers it in conjunction with GCM for the data cipher. FF also offers only the 3 most popular "curves" from SECG/NIST: P-256, P-384, P-521. It might support others but it doesn't say so.
What ECDHE requires (any server): to support any ECDHE ciphersuite in SSL/TLS you do NOT need an ECDH cert, much less a DH cert. You need two things in the server:
a privatekey and cert that allows signing, either RSA with a size acceptable to the client or ECC(ECDSA) on a curve supported by and acceptable to the client (and the server), with the cert issued by a CA/chain trusted by the client (and not expired or revoked etc.) and any chain cert(s) either supplied by the server or dynamically available or both.
Best practice since the beginning of 2014 is that RSA must be at least 2048 bits, ECC at least 224 bits, and signing hash at least SHA-256. FF apparently doesn't check RSA-2k yet, and it doesn't enforce SHA-256 yet but it does give a warning in its optional console(s), but you should expect it probably will enforce these before very long for "proper" (CA-issued) certs; whether it will do so for selfsigned certs that necessarily must be manually trusted is less clear to me. By choosing the "top 3" ECC curves FF implicitly enforces more than ECC-224.
ECDH parameters -- not a key and definitely not a cert, but only parameters, and not DH parameters either. ECC parameters are usually called a curve because that's the interesting part although the actual parameters also include the base point, order and cofactor. The curve used must be supported by and acceptable to the client (and the server).
What OpenSSL server and selfsigned needs is more specifically:
privatekey and cert that allows signing either RSA or ECC (ECDSA). Lots of browsers and clients besides FF like P-256 so if you use EC use P-256 unless you have a strong reason otherwise. As noted in the first part of my answer, make sure you use the "named" form of the curve not the "explicit" form. If you don't include the KeyUsage extension, it allows signing implicitly; if you do include KU, the value must have the digitalSignature bit set. Any selfsigned cert (for any ciphersuite and keyexchange not just ECDHE) will have to be manually trusted in FF.
"temporary" (ephemeral) ECDH parameters. You can:
SSL_[CTX_]set_tmp_ecdhto set a specific curve for future sessions (unless changed);
SSL_[CTX_]set_tmp_ecdh_callbackto set a callback function which will be called for each session to select a curve (unless changed);
- or in 1.0.2 (released Jan. 2015) call
SSL_[CTX_]set_ecdh_autoto have libssl automatically choose a curve suitable for each session.
As above use P-256, by calling
EC_KEY_new_by_curve_name(NID_X9_62_prime256v1), unless you use auto. (
EC_KEY_newdoesn't generate a key; it creates an object containing the curve which will be used to generate the key in libssl.)
SSL_k (and _a) bits cannot be in a cert. They are used only within OpenSSL as attributes of ciphersuites. There are some different values sometimes used in some certs to represent information related to but different than the SSL_k and SSL_a bits. But not for SSL_kPSK or SSL_aPSK; those never correspond to anything in a cert, because PSK never uses certs. I'm not sure what you actually did here, but the above description is what is correct.