Temporarily setting aside public key crypto specifically, I want to expand on a tangent from @fgrieu's answer.
Frankly, before digital computers the need and use for what we now consider strong encryption was minimal. Asymmetric encryption is particularly vulnerable but not alone in this regard.
Functionally, breaking crypto is about effective keyspace. To put that in to practice there are three steps:
Determine the method used. One of the desired properties of an encryption scheme is signature reduction. It goes without saying that you want to avoid cryptographic weaknesses that leave artifacts in the cipher, but going further than that you also want to avoid leaving leaving indication of what cipher was used in the first place.
A perfect cipher would be indistinguishable from random noise for anyone not in possession of the full decryption settings, or to put it another way, the only way to validate a cipher's integrity is to decrypt it. If you can't tell which of 10 ciphers I used then I've multiplied the work you need to do to break it by 10.
Identify the weakest points. No cipher should be assumed perfect, at best we can say there are no known attacks. Trying to find attacks on the cipher itself is an important part of cryptanalysis, both from the desire to break the cipher and for proving its strength.
Equally, people's application cryptography can't be assumed perfect. People tend to be dumb about these things. The more human involvement there is in the process, the more chance they'll mess something up and make your life easier by weakening things.
Compute through whatever keyspace is left. Unless you get really lucky, chances are you've still got more than 1 permutation left to try. So you'll need to try decryption with each permutation left in your keyspace. Maybe you'll find an working set of inputs after 1%, maybe it'll be at 99%.
So here's the thing. You don't need an especially large keyspace when the computers are people with paper, that's why combination locks (kinda) work.
From the actual key perspective, a 3 digit combination has a keyspace of 1000 combinations. From my experience with a particular type of 3 wheel combination lock I figure about 20-30 minutes on average to get the code purely by exhaustive search. Finding a good weakness or mechanical speed up will reduce the effort needed by at least an order of magnitude and often far more.
Even as recently as WW2, Enigma was sufficient. Actually, properly used Enigma is still hard to crack. Ciphers that require fast computers to use effectively aren't practical without said computers. Ciphers based on mechanical implementations that are hard to optimise mathematically or computationally, those are the ones we wanted for a long time.
So, why are asymmetric ciphers particularly vulnerable? Well it depends on what your goal in choosing it is.
In one case, asymmetric encryption is good for adding an authenticating factor to the message. With symmetric encryption possessing the key means you can forge the messages as well as intercept them. If you're using an asymmetric scheme you can only impersonate or receive the key you have. Yeah, this does work but you can achieve the same thing in other ways without the drawbacks of using 1:1 messaging schemes when broadcasting information.
In the other case, the public part of public key refers to using the ability to publish the key to enable a more secure exchange of another (usually symmetric) key. Exactly how and what is exchanged in the process varies. The issue is that most of the benefit requires real time communication and extra work and using separate mechanisms for the sending and receiving portion.
A modern analogy for using PKI for the first purpose is PGP and its family, where asymmetric encryption and PKI is used both for authentication and securing messages. You can maybe follow why it's less effective by considering a mailing list where everyone is sending PGP encrypted messages.
The second purpose, is what most people think of when talking about Public Key encryption... it's what makes HTTPS and SSH work. Imagine if, every time you wanted to make a phone call, you had to read out a 10 digit number, write one down, punch the new one in to the phone, and then hope you can both understand each other.
The biggest problem with PKI specifically, going back to the three stages I talked about above, is that to effectively perform stage 3 you ideally need to have done stage 2, and to do stage 2 effectively you need to have done stage 1. PKI, by nature and design, publishes not only part of the cipher settings but what cipher you're actually using. With computers in the picture you can compensate for exposing that information by just making the key space many many orders of magnitude larger. Still, if you can, it's always preferable to reduce how much you information you reveal to an adversary.
So, given the issues, and the fact that it only really becomes useful with the advent of trust infrastructure, electronic computation, fast high quality telecommunication and key exchange algorithms... it's actually not surprising that it didn't become notable earlier, evidenced by the difficulties mentioned in the answer from @ella-rose.