Some people say a symmetric one is faster than an asymmetric one, but is this that significant?
Yes. It is significant enough that most asymmetric systems are actually hybrid—they present an asymmetric external interface, but internally try to maximize their use of symmetric components.
The asymmetric primitives in common use (RSA, discrete logarithms, elliptic curves) are all potentially vulnerable to attacks if somebody manages to build a practical quantum computer. The symmetric primitives in common use are much more resilient to such attacks—it would possibly require going from 128-bit keys to a larger size, but that's it.
(See: post-quantum cryptography, for information on efforts to develop asymmetric primitives that will resist quantum computer attacks.)
Lack of need
Many applications just don't need asymmetric cryptography in the first place. Take, for example, storage encryption (as used, e.g., in modern smartphones) where the "sender" and "recipient" are in fact the same person but at different points in time. There is little or no need for asymmetric cryptography there, because any secret key, by definition, is held by both the sender and recipient.
Another example: encrypting a simple WiFi network (e.g., a home network) with a pre-shared key, where the key distribution problem is manageable. Entering a WiFi secret key manually into a handful of devices isn't a big logistical challenge.
Symmetric cryptography is much less hardware-demanding, which translates to more affordable costs. For example, two-factor authentication devices like RSA SecurID often use symmetric crypto simply because it's more affordable.