In CPA model, we assume Eve has access to an oracle who has the correct key and Eve can send query to the oracle to encrypt in CPA the data.
Assume this encryption algorithm is a plain ECB mode AES and eve knows about it, i.e., if Eve plug in a key to AES-ECB and apply the same data for encryption twice, Eve will be able to distinguish that the ciphertext is not random across two encrytions with the same key.
However, let's say Eve can gain a few ciphertexts that is encrypted with the real key from the cipher oracle and he knows the plaintexts corresponding to these ciphertexts. However, Eve cannot control the plaintext value. Then the key is removed from the cipher oracle and the cipher oracle will never encrypt anymore plaintexts.
But Eve can plug in any key Eve likes after the real key is removed. So do we still say such a system is subject to CPA attack? NOTE: his goal is to obtain information about the plaintext.