The Hamming distance is more effective when you suspect that the plaintext has been XORed with some repeating keystream. That's because XOR works at a bit level, as does the Hamming distance.
The Index of Coincidence is more effective when you suspect that the plaintext has been combined with some repeating keystream, where the combiner works character-by-character instead of bit-by-bit. For instance, the IoC will be more effective for a Vigenere cipher, or for a repeating-key polyalphabetic cipher. This is because those ciphers transform individual characters of the plaintext as a unit (not individual bits), and the IoC works at the character level.
If you like, you can think of the Hamming distance technique as effectively applying the Index of Coincidence at the bit level: the Hamming distance is effectively a variant of the IoC that works at the bit level, and that is especially effective when the combiner is the bitwise XOR operation. Therefore, if you know that the plaintext is combined with some repeating keystream using some bitwise operation (such as XOR), then you might use the Hamming distance (effective, applying the IoC at the bit level instead of the character level).
Of course, you can always apply both and see which works better on your particular cipher -- but the above discussion tells you which method you should expect will work better in any given circumstance, and why.