I assume you follow Kerckhoffs' principle so the attacker knows the padding scheme and derivation function so the answer is yes, it only takes a few seconds to decrypt and anyone can do it.
If he doesn't know these things, he can find them by trial and error (assuming he can get his hands on a valid ciphertext).
The IV can be sent in the clear so making it depend on the key reveals some information on the key. It should also be unique for each session so I'm a bit worried about it. It should be OK if the KDF you're using is non-deterministic, which implies it uses its own IV so the problem remains. See this questionthis question for more on IVs.
Edit based on comments: The cracking procedure is the following: $a$ is the number of all modern ciphers. Let's set this to 100. $m$ is the number of modes per cipher, set it to 6. $k=100$ is the number of key derivation functions, $p=100$ are the padding schemes. The values are arbitrary. So we have $(p*k*m*a*c)/n = 6*10^6$ which equals to approx. 69 days with $c=1$ second and $n=1$ processor.
Your adversary will solve the problem in $69/2=34.5$ days. Not as practical but definitely feasible, especially if you throw extra processors at it. This solution is completly brute-force, it makes no attempt at distinguishing and pre-eliminating ciphers.
Since non-indistinguishability is a business requirement (really?) you could just get away with using ECB instead of any other mode.