This is something I tend to disagree somewhat with Colin Percival on.
You should use Encrypt-then-HMAC if and only if you can get it right. The biggest pitfall is using a short-circuiting string comparison versus a constant-time string comparison. Given the former, people can use timing attacks to forge valid HMACs for arbitrary ciphertexts. With an encryption mode like CBC, this could be used to read the contents of encrypted messages. Other pitfalls include accidentally implementing Encrypt-and-MAC or MAC-then-encrypt, which have historically had vulnerabilities related to their construction.
On the other hand, EAX and GCM modes provide this feature transparently. There are some concerns about potential side-channel attacks against dedicated AEAD modes, and there is also the conceptual issue of potentially passing unauthenticated packets to your decryption algorithm. You, of course, have to get details like unique IVs correct, but you also have this requirement for Encrypt-then-MAC.
That said, potential flaws in EAX and GCM implementations can be fixed for many people at once with patches to the relevant libraries. Flaws where non-cryptographers naïvely compare HMACs are epidemic, and even high-profile projects like Google's KeyCzar have made this simple mistake. If you absolutely know what you're doing, Encrypt-then-HMAC is a provably secure construct. If you don't know what you're doing, something like EAX or GCM may be found to have weaknesses, but they are assuredly better than implementing something yourself.
Edit: I may have made Encrypt-then-HMAC sound easier than it actually is. One other major pitfall of Encrypt-then-HMAC is not HMACing enough information, or not HMACing the information correctly. The HMAC must include the ciphertext, the additional authentication data, and the initialization vector. It probably must also (perhaps someone else can confirm or deny this) include a descriptor to identify the encryption algorithm used. To pass these fields into the HMAC, you must use a format that unambiguously delineates fields. The easiest way to do this is to prepend a 4-byte big-endian length before any variable-length field (usually just authentication data and ciphertext), but doing this for all fields is probably a good idea. If you don't HMAC all this information, an attacker can modify any of the data not included. If you don't HMAC them unambiguously, an attacker can manipulate message boundaries, which could allow him to conduct an exploit.
Edit 2: Another detail of Encrypt-then-HMAC I've forgotten. Hopefully you can see how hard this is to get right. Ideally, you should never reuse the same key across security contexts. I'm unaware of any attacks that would result from using your encryption key as the key for the HMAC, but best practice is to use a different key for encryption and authentication. One simple approach (assuming a 256-bit encryption algorithm and 256-bit HMAC) is to use a "virtual" 512-bit key. Use the first half for encryption and the last half for authentication. Another approach is to use a 256-bit key, but pass it through PBKDF2 with a 512-bit output; use this as before: split it into two parts, use one for encryption and one for authentication.
Edit 3: I asked a highly relevant question on the security StackExchange a few months ago. Thomas Pornin's detailed answer may be helpful.