In the setting of Encrypt-then-MAC, do I need to include the IV in what I'm HMACing, or is authenticating just the AES-encrypted data sufficient?
In short: You must authenticate the IV. Which particular attacks apply if you don't depends on the block cipher mode; I will give two common examples.
In CTR mode, an attacker who fiddles with the IV can forge authenticated messages, but the content of the corresponding plaintext is beyond his control (since he doesn't know the key). Depending on the encrypted data, this may or may not be a real problem (as random binary garbage may be caught by subsequent validations in the protocol), but it certainly is an unnecessary security risk in any case.
CBC mode breaks horribly, as the following diagram shows: Clearly, the IV is exclusively XORed into the first plaintext block after decryption and discarded afterwards, enabling an attacker to introduce arbitrary differentials in the first block without changing anything else by XORing them into the IV. Depending on the payload's structure, this could, for example, be used to toggle some important flag in a protocol header or change a length field.
The Encrypt then MAC is done in general in order to be sure to decrypt into the correct plaintext, without risking of parsing a non-authentic plaintext message.
If you don't MAC the IV, then Mallory (attacker that can tamper with messages as a man-in-the-middle) can modify the IV and your MAC will be still validated as good.
So you will decrypt into an something different than the message, because the change in the IV will change the decrypted text.
If, instead, you HMAC also IV then you avoid the issue.
Therefore it is not safe to not process the IV during the HMAC computation.
EDIT: due to valuable comments by D.W. and poncho