The problem with passwords or, more succinctly pass phrases, is that they are often relatively easy to guess. Password strength - or rather the lack of it - is a common problem. The strength of a password is not such a big deal if there is some kind of control over how the password can be used (e.g. a PIN is relatively easy, but you are only allowed 3 tries). However, if they are used to generate the key then the ciphertext is commonly believed to be accessible by an adversary, leaving the password guessing vulnerable to an uncontrolled offline attack.
Cryptographic hashes have been designed to be not just one way, but also to be rather fast. That simply means that for an adversary it is easy to test many inputs to derive the correct key. Generally it only takes very little time to test a key as well, since it is not required to decrypt a large amount of data with it; if the ciphertext is large then the adversary can still just decrypt a small part of it. So an adversary has to do a limited amount of work to find a key.
The iteration count - or more generally - the work factor is there to slow the adversary down. Any work that you do once for each password, the adversary has to do for each guess. That still only provides an additional amount of work compared to the situation with a cryptographic hash, but beggars cannot be choosers. So this is what passwords hashes or Password Based Key Derivation Functions have been designed to do: "strengthen" or "stretch" the password in such a way that they are relatively harder to guess.
As said, PBKDF's only provide limited protection against adversaries. Moreover, since ciphertext can be stored, the protection diminishes when faster computers become available. Attacks can also be very easily parallelized. So I would very much try and avoid password based encryption all together.
There are some ways of mitigating the problem of course. For one, you can use a password manager and a very complex password that has a similar strength of a secret key (say 96 bits or higher). That way you may not need a PBKDF to strengthen the password (although entering the password may become more cumbersome).
An other way of mitigating this is to encrypt a private key instead. Now you access to the private key to decrypt the ciphertext. You don't need access to it during encryption.