Can we get more precise than this?
Actually the "randomness" of the resulting values is not the blocking issue with composite moduli for DH.
What is much more severe is the fact that if you know the factorization of the modulus $n=pq$, then solving $g^x\bmod n$ is essentially the same as solving $g^x\bmod p$ and $g^x\bmod q$ both of which are substantially easier than solving the problem for $n$ itself (using eg GNFS). Afterwards you can cheaply recombine the solutions with CRT.
In particular note that using a 2048-bit safe prime for DH yields a secure (unauthenticated) key exchange (with effort needed of about $2^{112}$ to break it) whereas if you were to use eg balanced factors that would drop to solving two 1024-bit discrete log instances, both of which need about $2^{80}$ work (so $2^{81}$ in total). This is considered feasible if you have the right amount of funding (eg nation-state level). Obviously if you use unbalanced factors the run-time of attacks is dominated by the larger factor and at that point you may just do away with the smaller one and save yourself the computation time (for your normal exchanges).
If you don't know the factorization, things become harder, but so does uncertainity. You have to trust that whoever generated the composite modulus actually doesn't know the factors (and thus is able to much more easily break the security). Also it allows for parameter replacement attacks which in turn do allow such attacks. This is not a theoretical attack. And in fact with such a parameter replacement attack you could just sneak in a value that is a composite of 4x 512-bit primes, nobody could factor that (easily) but with about a week of pre-computation every discrete log instance with that modulus could be solved in a matter of seconds.
So TL;DR: By using composite moduli you only gain a lot of headaches and neither speed nor security.
If the above is too complex try the following:
We use primes for the modulus in Diffie-Hellman because it gives us no speed nor a security advantage not to use them. In fact quite the opposite, if we used composites, we could split the relevant hard problem of discrete logarithm along the factorization and solve both sub-problems in parallel. Even worse each of these instances is significantly easier to solve due to the large reduction due to the solving algorithm's complexity growing really fast with the problem size. If we don't know the factorization we'd have to trust the generator of the modulus not to exploit the above weakness against us and attackers may replace our parameters with theirs, giving them backdoor access to the data.
