Since you do not describe why TLS Handshake and IKE are appropriate in your situation, and as long as you don't describe your situation, it's hard to really help you. Also, you haven't stated if it's only IKE that's not appropriate, or if that also includes IKEv2 (which improved the IKE protocol). Therefore, I'll simply assume you meant both.
As an alternative to IKE, the first thing that comes to mind is KINK (Kerberized Internet Negotiation of Keys). It's a protocol defined in RFC 4430. It is used to set up an IPsec security association (SA), similar to Internet Key Exchange (IKE), utilizing the Kerberos protocol to allow trusted third parties to handle authentication of peers and management of security policies in a centralized fashion.
As said — as long as you don't describe your situation, it's hard to pinpoint what might suit it best.
It could well be that your situation might be solved with RFC 6539, as IBAKE provides an Identity-Based Authenticated Key Exchange.
Or maybe "augmented PAKE" is a valid option for your situation. That's a variation applicable to client/server scenarios, in which the server does not store password-equivalent data. This means that an attacker that stole the server data still cannot masquerade as the client unless they first perform a brute force search for the password. Examples include: AMP, Augmented-EKE, B-SPEKE, PAK-Z, SRP, and AugPAKE (RFC 6628).
It's hard to tell when not knowing your situation. But I would check out KINK first… as that should be a close-to-perfect match, being an alternative to IKE.