There are two reasons:
Reason one is that the ECDH shared secret is not equidistributed; not all values are possible. In particular, and $x$ that is not a possible solution to the elliptic curve equation cannot happen at all.
Things that use the shared secret (such as AES) are generally assumed to have uniform keys; that is, all keys are possible (and are equiprobable); this isn't possible when using the ECDH shared secret directly.
In practice, it probably doesn't matter; it's hard to conceive of an attack that becomes practical because half of the possible $x$ coordinates are impossible.
Reason two is that not all bits of the ECDH shared secret are actually independent. If the part of the shared secret bits start going into the $y$ coordinate, then if the attacker somehow gets the $x$ coordinate bits, he gets the $y$ coordinate bits for free (well, one of two possibilities).
We try to avoid revealing some of the bits of the key; however having some bits imply others isn't what we want. A proper KDF (key derivation function) avoids this possibility.
Note: reason two doesn't apply to all elliptic curves; for example, with Curve25519, we don't explicitly compute the $y$ value, hence that's not available as part of the shared secret. However, it is present for other curves.