Or the elliptic curve and the generator point are known to everyone?
Yes, base point is public parameter in elliptic curve cryptography. Suppose that, your base point is G and your private key is k. In this case, your public key would be kxG. Calculating kxG (by multiplying k and G) is fast whereas extracting k from public kxG is slow. In other words, you can calculate kxG in seconds wheras extracting k from kxG last almost infinite with brute force attack. There are approaches to calculate kxG fast such as double and add method (aka binary method).
That's the power of the discipline.