How is the shared secret generated in IPsec using DH key generation?

I have doubt like when in the IPsec we use a DH group for key exchange. It takes a prime number and $$g$$ value from group bit.

How does the key exchange payload and nonce help to generate the shared secret in the he message 3 & 4 exchange?

This is presumably an IKE (RFC 7296) question, rather than an IPsec (which deals with the encryption of data traffic).

I wasn't quite sure if you're asking "how does IKE use DH" (which the RFC spells out), or "how does DH work" (which the RFC assumes you already know); the below tries to answer both (and for simplicity, I assume a finite field group - elliptic curve groups are similar, but usually use slightly different notation).

In message 1, the initiator picks a random value $$x$$, computes a value $$I = g^x \bmod p$$, and sends that (along with other data, such as which group he proposes, that is, which $$g, p$$ to use) to negotiate the SA.

In message 2, the responder picks a random value $$y$$, computes a value $$R = g^y \bmod p$$, and sends that (along with other data, such as those values of $$g, p$$ are acceptable) to negotiate the SA.

After the responder sends message 2 (and the initiator receives it), both can compute the shared secret; if you're using a finite field group, then the initiator will compute $$R^x \bmod p$$. Similarly, the responder will compute $$I^y \bmod p$$. By the magic of DH, both these computations will result in the value $$g^{xy} \bmod p$$; this value is believed to be hard to reconstruct from the values $$g, p, I, R$$.

Once IKE has this common secret value, then both sides will compute:

$$SKEYSEED = prf(Ni | Nr, g^{xy} \bmod p)$$

(where $$prf$$ is the negotiated pseudorandom function, and $$Ni, Nr$$ are the IKE nonces exchanged in the initial message), and then will expand $$SKEYSEED$$ (along with the nonces and the SPIs) into symmetric keys used to protect the IKE traffic (and the generate keys for child SAs); this is spelled out in section 2.14 of the RFC.