RFC 5280 is the relevant standard for X.509 PKI certificates. Per section 4.1.2.7 of that standard:
The object identifiers for the supported algorithms and the methods
for encoding the public key materials (public key and parameters) are
specified in RFC3279,
RFC4055, and
RFC4491.
RFC 3279 specifies the following public key types:
- RSA
- DSA
- Diffie-Hellman Key Exchange (DHKE)
- Key Exchange Algorithm (KEA)
- Elliptic curve DSA (ECDSA)
- Elliptic curve Diffie-Hellman (ECDH)
RSA 4491 adds the following public key types:
- GOST R 34.10-94
- GOST R 34.10-2001
RFC 4055 adds some signature and other algorithms, but no additional key types.
Finally, section 4.2 of RFC 8692 adds that "certificates conforming to RFC5280 can convey a public key for any public key algorithm", so other types seem possible, although obviously certificates containing more exotic keys may not be interoperable.
Specifically for public-trust TLS certificates, the CA/Browser Forum Baseline Requirements restrict the key types to RSA and ECDSA (section 6.1.5), with additional restrictions appropriate to the type. For TLS more generally, the public key in the server certificate should be compatible with the selected key exchange algorithm.