Asymmetric keys have to be much larger than symmetric keys because 1) there are less asymmetric keys for a given number of bits (key space), and 2) there are patterns within the asymmetric keys themselves.
To compare, consider that the ECRYPT II recommendations on key length suggest a 128-bit symmetric key is as strong as a 3,248-bit asymmetric key, and these equate to a 256-bit Elliptic Curve Diffie-Hellman (ECDH) key. Both key lengths (128-bit and 3,248-bit) are generally recognized as providing long term protection of data (until 2040).
For further comparison, NSA Suite B Cryptography (unclassified algorithms) specifies both the 128-bit symmetric key space (for encryption) and the 256-bit ECDH key space (for signing and key exchange) as adequate to protect classified data upto Secret.
Typically, symmetric-key algorithms in common use are designed to have security equal to their key length. However, no asymmetric-key algorithms with this property are known; elliptic curve cryptography comes the closest with an effective security of roughly half its key length. (source: Wikipedia)
In a perfect world, the idea is that the only way to break into a network connection or a data store secured with a symmetric cipher is to try all the keys. A 128-bit key space means there are only just a mere 340,282,366,920,938,463,463,374,607,431,768,211,456 possible keys that could be used.
Consider trying to test all these keys for a 128-bit AES encryption using the special AES instructions added to the latest Intel microprocessors. These instructions are designed to be very fast and according to Intel's own data decrypting a block of AES encrypted data would take 5.6 cycles on Intel i7 Processor with 4 cores.
Put another way, that processor could try out one key on one block of data in about 1.7 nanoseconds. At that speed it would take it about 1.3 * 10^12 * the age of the universe to check all the keys (you'd probably only have to check half before finding the right one so divide that incredibly long time by two).
So, breaking 128-bit keys by brute force isn't terribly practical. (Breaking 256-bit keys is even less possible.) For symmetric ciphers, keys of these lengths make sense.
Now, unlike a symmetric cryptosystem, asymmetric cryptography works by having two different keys (one for encryption and one for decryption), which are related by some mathematical process.
For example, in the popular RSA scheme used with SSL/TLS the public and private keys consist in part of the product of two large prime numbers, and so making an RSA key starts with picking two random primes. The security of RSA relies (in part) on the fact that it's easy to choose two random prime numbers, but it's very hard to discover what they are when just given their product.
Suppose there are two prime numbers picked at random called p0 and p1. Part of the RSA public (and private) key is called the modulus and it is simply p0 * p1. If an attacker can factor the modulus into p0 and p1 they can break RSA because they can determine the private key. Mathematicians believe that it is very hard to factor a product of two primes and the security of web transactions relies, in part, on that belief. (The "hardness" of integer factorization has not been proven. It's presumed difficult but there's no real formal proof of it.)
Typical RSA key sizes are 1,024 or 2,048 or 4,096 bits. That is the number of bits in the modulus. For each there will be a pair of primes of about 512 bits or 1,024 bits or 2,048 bits depending on the selected key size. Those primes are chosen by some random process (highlighting the importance of cryptographically secure random number generators).
Just as in the case of symmetric keys, attacks on say 2,048-bit RSA are based on trying out all keys of a certain size, but unlike the symmetric key scheme not every 2,048-bit number is an RSA key (because it has to be the product of two primes).
So, although the key space is larger there are actually fewer possible RSA keys for any given number of bits than there are for the same symmetric key size. That's because there are only so many prime numbers of that size and below. The RSA scheme can only use pairs of prime numbers, whereas symmetric schemes can use any number at all of the same size.
Any time there's a pattern in a key it represents a weakness in the cryptosystem. For example, in a perfect world, people would create and memorize completely random passwords. Because they don't, there are patterns in their passwords which can be guessed or broken without having to try every possible one.
RSA keys have a distinctive pattern: they are the product of two prime numbers. That provides the weakness, and it is best exploited by the General Number Field Sieve. In the symmetric key case there are no such patterns: the keys are just large randomly-chosen numbers.
Source: Why some cryptographic keys are much smaller than others