Shared secret resulting from the Diffie-Hellman step is a mathematical object; namely, the X coordinate of a curve point. It is a value in a non-binary range; moreover, it is indistinguishable from randomness only up to the security against discrete logarithm, i.e. about 128 bits. Thus, it is at least debatable that parts of the key might be guessable from the public DH values.
In other words, there are at least 128 bits of entropy ("entropy" here used as a loose meaning for "resistance to guessing attacks") in the 256-bit output of the DH, but you don't know really where. If you extract, for instance, the first 128 bits with basic truncation, then you don't know if you got your 128 entropy bits, or less, possibly much less.
No exploitable bias is really known at that point, but we know enough to be wary. Hashing the DH output is a simple and safe way to gather all the entropy and spread it around as needed, hence the recommendation.