This question is based on a false premise, namely, that the difficulty of breaking cryptography (involving a given key type) depends only on the size of the key. This is not how cryptography is broken. The weak point in cryptography is usually flaws in the implementation.
For example, one way RSA can go wrong is if the key was not generated correctly. Cases of keys that were generated using a random generator that was not properly seeded are depressingly common. For example, Debian shipped a broken version of OpenSSL for two years which resulted in there being only a tiny number of possible SSH keys. Surveys of SSH and TLS public keys have shown that a small, but significant proportion of servers have keys that were generated with insufficient entropy or with a flawed process, resulting in keys that can be broken easily. (An RSA private key is determined by two (pseudo-)primes, and the public key is their product. If two private keys share a prime, it's trivial to spot this from the public keys $p \, q_1$ and $p \, q_2$.) Linux has had a steadfastly broken interface to random generation, historically giving the choice between an interface that blocks needlessly and one that is insecure in one specific but critical moment (just after the first boot of a system) (modern kernels have a good interface in the form of the getrandom system call, but because it's harder to use than the historical interfaces, a lot of software relies on the historical interfaces). In addition to the RNG problem, there is flawed key generation code out there, for example not using enough iterations when testing primality (which is a probabilistic test), or using computational shortcut that make the private key easier to calculate from the public key than brute force (such as generating primes that are close to each other, which can be faster, but is insecure).
A larger key size can help mitigate against a broken key generation process. For example, generating a larger key will request more material from the RNG and this can help if the RNG is not well-seeded at the beginning but receives additional entropy over time. Generating a larger key can also help if the code takes shortcuts, because it may compensate the loss of security from these shortcuts. But not all flaws can be mitigated that way. When the RNG is not seeded properly and only a small number of keys are possible at all (as in the Debian-OpenSSL case, or in the case of many embedded devices without a hardware RNG), no increase of the key size will help.
In addition to the key generation process, another source of weaknesses is the use of the key. Asymmetric (and symmetric) cryptography is prone to side channel attacks, where properties such as the timing of certain parts of the computation leaks information about the key, or about the data that the key is protecting. A larger key can help against some attacks, because many side channel attacks need to observe a large number of calculations in order to extract enough information to reveal the key or the data.
Asymmetric encryption is also prone to protocol attacks. Asymmetric decryption is tricky because it combines the private key with an input that cannot be trusted since it could have been produced by anyone. This can allow oracle attacks, where the attacker sends an invalid ciphertext, and based on the message or the timing of a decryption error, learns some information about the key. The general principle of this form of attack on RSA was published in 1998 by Bleichenbacher and was still a concrete problem 20 years later. Modern communication protocols avoid asymmetric decryption in favor of key establishment protocols where, rather than encrypting a session key with a symmetric key, both sides arrange to derive the same session key from different computations that give the same result. But RSA decryption is still used in the wild in TLS up to 1.2 (this is deprecated, but still used by “middleboxes” that intercept and filter TLS traffic in enterprise networks).
if a 2048 bit key is a square, would anyone actually bother checking this?
Well, yes, of course. It's very easy to check. It's also easy to factor the key if $p \approx q$, and it's one of the obvious things an attacker would try.
Where is the point where nobody will even try to crack it - with the result that some stupid mistake making a particular code insecure would never be detected, because nobody tries?
There is no such point. If you generate a key without a proper RNG, or if you store your key insecurely, or if you use an implementation that doesn't properly protect against applicable attacks such as side channel leaks and glitches, or if you use a flawed protocol, your key may be vulnerable no matter what its size is.
“Nobody will even try to crack it” is not a useful concept. It's based on the assumption that attackers are stupid and work in isolation. They aren't and they don't. Where there's a possible attack, all it takes is one expert to write a tool to perform a sophisticated attack, and sooner or later a conveniently packaged version of this tool will be available for free or for sale.
