I am quite new to the study of elliptic curve cryptography and as such I might be asking something with a mundane solution, but I can't easily find such a solution online. My understanding of ECC is that you can generate a private key (some integer $k$), a starting point on the curve ($G$), and a curve equation, and then generate a public key through finding $kG$. My understanding is then that your computer would perform however many operations are required to find $kG$ (if $k$ was 16 then that would be four operations).
With this data the starting point $G$, the curve equation, and the public key is made public. What I am wondering is why can't an attacker try to find out what the private key $k$ simply is, take the starting point and perform operations until they reach the public key and as such know what $k$ is? Is it based in the fact that the sender only needs 4 operations to calculate $kG$ whereas the attacker would need 16 operations (for the given example)?