Be careful when comparing key sizes. Different algorithms require different sized keys to achieve equivalent security. There are significant differences between symmetric block cipher key sizes and asymmetric (public/private) key sizes; and among asymmetric algorithms there are big differences, too.
Symmetric ciphers, such as AES, DES, Blowfish, etc., have key lengths that are usually directly equivalent to brute force efforts. To guess an AES 128 bit key requires an average of 2^127 guesses. To guess a 56 bit DES key requires an average of 2^54 guesses. An algorithm that doesn’t hold true to this is considered weak, and shouldn’t be used.
Asymmetric ciphers are based on different types of math problems. RSA is based on factoring the product of two prime numbers. Factoring algorithms are much more efficient than trying every possible number. For example, you wouldn’t try any even numbers, because they’re all divisible by 2. You wouldn’t try any number ending in 0 or 5, because they’re all divisible by 5. So to get the equivalent protection of 80 bits of guessing, mathematicians have calculated that your RSA key needs to be about 1024 bits long.
Similarly, Elliptic Curve Cryptography uses a different hard math problem: finding the intersection of points on a curve. The numbers required to achieve similar results are smaller than RSA factors, but larger than brute force. It may take an ECC key of about 160 bits to equate to the security of an 80 bit symmetric key.
Also, these difficulty factors are probabilities determined by mathematicians based on what they know today. Tomorrow, someone may come up with a new, more efficient factoring algorithm, rendering these key sizes obsolete.
So to reiterate: a 1024 bit RSA key is much, much weaker than a 128 bit AES key.
Within a single algorithm, yes, larger key sizes are harder to crack. This is where you use published information to understand the expected brute force capabilities, and select a key size that is resistant, but not so large that it drags down performance. An 8192 bit RSA key is not going to be brute forced any time soon, but do you really want to build a server farm just to do TLS key exchanges?