Blocks ciphers work by applying operations to an $n$-bit block so as to achieve confusion and diffusion. In short, a good block cipher should "mix" the bits of the plaintext and key as thoroughly as possible, so that it becomes practically impossible to recover the key or decipher unknown ciphertext.
Now, to achieve confusion and diffusion, there are a few basic operations in common use: xor, addition modulo 2^32 or 2^64, bit rotations, (small) table lookups, and a few others. The thing that all these operations have in common is that they're all operations a normal CPU can perform very quickly. Hell, a Core 2 CPU can perform 6 billion of adds or xors per second!
Public-key cryptography, on the other hand, relies on the existence of trapdoor functions. A trapdoor function is a function that is very hard to invert, unless one is given some special information, at which point it becomes easy. Coming up with good trapdoor functions is no easy task, and the ones we know (and trust) today are mostly based on number-theory. The most widely known, RSA, allegedly relies on the hardness of inverting the function
$f(m) = m^e \pmod n$,
where $n$ is composite and of unknown factorization. Inverting this function is very hard, for large enough $n$. The "large enough" bit is the key here---in the case of RSA, $n$ must be at least 1024 bits long to be safe, due to the power of the number field sieve, and thus operations (read: exponentiation) must be done modulo at least a 1024-bit number.
This is exactly why RSA is much slower than, say, AES: arithmetic of numbers much larger than the CPU's natural word length is slow, and exponentiation requires $O(\log e)$ multiplications for an exponent $e$ (RSA-1024 requires on the order of 1024 multiplication modulo a 1024-bit number). For example, the eBACS project reports 1640960 cycles for a single RSA-1024 decryption, in a top-of-the-line Sandy Bridge processor. The situation is ameliorated by elliptic curve cryptography, which require smaller key to achieve the same level of security; it is still slower than symmetric ciphers, though.