The advantage of asymmetric encryption is not security, it is capability. Asymmetric crypto provides completely different capability that you can't get with symmetric crypto. With symmetric encryption, you have to establish a key a priori. With asymmetric crypto, you can establish a key on the fly (granted you have to have trust in the public key, that it belongs to the party you want to interact with). With asymmetric crypto you often have digital signatures, establishing authenticity of data.
You mention the primary advantage of symmetric crypto in your question, speed.
So, to your questions:
Why couldn't I do the same with Asymmetric encryption?
You could. You would first have to define how you are going to encrypt large amounts of plaintext. Do you build a > 1TB modulus? Absolutely not. Do you establish some sort of mode of operation? Sure, but then you have to prove security.
You also will likely want integrity checks like you would get with authenticated encryption or HMAC. How do you do that?
what are the practical limitations of using Asymmetric encryption for bulk encryption?
Speed and the lack of trusted, well studied standards.