How is the shared secret generated in IPsec using DH key generation?

Question

I have doubt like when in the IPsec we use a DH group for key exchange. It takes a prime number and $g$ value from group bit.

How does the key exchange payload and nonce help to generate the shared secret in the he message 3 & 4 exchange?

Answer 1

This is presumably an IKE (RFC 7296) question, rather than an IPsec (which deals with the encryption of data traffic).

I wasn't quite sure if you're asking "how does IKE use DH" (which the RFC spells out), or "how does DH work" (which the RFC assumes you already know); the below tries to answer both (and for simplicity, I assume a finite field group - elliptic curve groups are similar, but usually use slightly different notation).

In message 1, the initiator picks a random value $x$, computes a value $I = g^x \bmod p$, and sends that (along with other data, such as which group he proposes, that is, which $g, p$ to use) to negotiate the SA.

In message 2, the responder picks a random value $y$, computes a value $R = g^y \bmod p$, and sends that (along with other data, such as those values of $g, p$ are acceptable) to negotiate the SA.

After the responder sends message 2 (and the initiator receives it), both can compute the shared secret; if you're using a finite field group, then the initiator will compute $R^x \bmod p$. Similarly, the responder will compute $I^y \bmod p$. By the magic of DH, both these computations will result in the value $g^{xy} \bmod p$; this value is believed to be hard to reconstruct from the values $g, p, I, R$.

Once IKE has this common secret value, then both sides will compute:

$$SKEYSEED = prf(Ni | Nr, g^{xy} \bmod p)$$

(where $prf$ is the negotiated pseudorandom function, and $Ni, Nr$ are the IKE nonces exchanged in the initial message), and then will expand $SKEYSEED$ (along with the nonces and the SPIs) into symmetric keys used to protect the IKE traffic (and the generate keys for child SAs); this is spelled out in section 2.14 of the RFC.