Are their values fixed for each DHE group ?
Yes, by convention a named group includes at least $(g,\mathbb G,q)$ with $\mathbb G$ being a description of the group parameters (i.e. $(p,a,b)$ for prime elliptic curves or $p$ for FFDH), $g$ being the generator and $q$ being the generator's order.
How do the communicating parties using Finite Field DHE agree on the values of the g and p variables ?
The flow in TLS 1.3 currently is:
- Client sends (in supported_groups) a list of 'names' (really code numbers) of the groups it is able and willing to use, which may be any combination of FFDHE and ECDHE, and optionally (in key_share) newly-generated public keys for some or all of these groups; it may send empty key_share if it wants to see the server's choice first, or if it is (also) offering and would prefer PSK-based session resumption.
- If xxDHE is selected (i.e. except when the client proposes and the server agrees PSK-only resumption) the server checks in key_share for a group it also is able and willing to use, and if found sends a generated public key in that group. If not, but the server finds an acceptable group in the client's supported_groups, it sends HelloRetryRequest specifying that group (but not yet any public key). If neither, the server aborts the handshake.
- If requested, the client sends a new ClientHello with a public key for the group specified by the server (but other changes strictly limited as per 4.1.2) and the server responds with its public key in that group (no change permitted).
In TLS 1.2 and below the flow depends on whether both systems implement RFC 7919, which was published many years after most TLS 1.2 implementations were written, much less lower versions:
- With RFC 7919, the client indicates in supported_groups, which originally was called elliptic_curves, the named groups it supports, either FF or ECC. There is no indication of support for other (arbitrary) FF groups, because that was the default long before this extension was created. For ECC groups/curves, previously (in RFC 4492) there were code values to indicate arbitrary/unnamed curves over Fp or F2^m in the Weierstrass form defined by X9.62 and 63, but this was rarely if ever implemented or used and RFC 8422 deletes this option.
- The server for FFDHE always sends the p,g values, plus its public key (and a signature), in ServerKeyExchange in a format defined long before even RFC 4492, whether those values correspond to a named group or another, arbitrary group. An arbitrary group could be generated by the server or its admin, or imported from almost any source. For example, the Logjam researchers found a large number of public servers using one FFDHE group that was coded into some versions of Apache httpd.
- The client either accepts the server's choice of group and generates and sends their public key in that group, or aborts the handshake. For example, especially in recent years, some client implementations abort if the server sends a group with p too small to provide the desired security level. (Java makes this a runtime-configurable option: jdk.tls.disabledAlgorithms contains "DH keySize < $number".) But even an FF group that is large enough numerically may be weak for less-obvious reasons; there are existing Qs about that.