If I have a very large input to hash with a hash function vulnerable to length extention and I am using a hash list, should I HMAC each part of the input or just HMAC the final hash of all the hashes ? If i do need to HMAC everything, should I use a different key for each hash ?
You can just use a hash tree with a pre-configured node size then only the last hash value would be vulnerable to a length extension attack. In that sense this is not that different from performing a HMAC over a single hash.
However, that's kind of besides the point. A length extension attack is only applicable for keyed hashes. As the hashes are not keyed yet, length extension attacks don't apply.
So yes, you can simply perform HMAC over the final hash (or final hash set if you are using a Merkle tree, and you want to minimize the depth of the tree). By that the hash value must remain the same as otherwise the HMAC verification will fail.
If the hash value remains the same it obviously is protected against length extension attacks; length extension attacks alter the final hash value while still representing a valid keyed hash.