It would depend on the hash function if this is more or less secure. If the hash is broken and allows collisions then the scheme is decidedly less secure.
For instance, if you'd use MD5 for the HMAC then you'd still be secure, as HMAC(k, m)
doesn't rely on collision resistance of the underlying hash function. However HMAC(k, MD5(m))
certainly does rely on it. So this latter scheme would be considered broken: an attacker can come up with m
and m'
that would be verified by the same HMAC if H(m) = H(m')
after all.
If - on the other hand - a secure hash function is used then it should be secure. Hash functions are often stacked on top of each other, e.g. in a Merkle tree. That's a known good application of hash functions and I don't think that there is anything wrong with that.
I don't see how length extension attacks would apply to HMAC(k, H(m))
. The key is not used in the hash calculation over the message, after all.
The security strength of the combination of HMAC and hash would very likely depend mostly on the hash. For sure, if HMAC uses SHA-256 and the hash uses SHA-256 then the security provided would top at 128 bits due to the collision resistance of the hash. Generally this would not pose an issue if the hash is strong enough, so I almost forgot to mention it.
Using SHA-512 or SHA3-512 instead of a weaker hash function may make a lot of sense...