If we encrypt our data with a longer SSH key which is protected by a shorter passphrase (with a decent entropy), then the limiting factor is entropy of the passphrase. So why do we use a longer key at all? We could simply encrypt the data with the passphrase.
You're not modeling what information the attacker has. The strength of the system only reduces to that of the passphrase if the attacker has a copy of the passphrase-encrypted private key. If they don't, then the strength of the system is the strength of the private key.
You're not supposed to let your passphrase-encrypted private key file fall into adversarial hands. The passphrase is providing a fallback in case that condition is violated.
Note that the passphrase is optional, and that there also exist non-passphrase mechanisms for protecting the private key. OpenSSH supports using smartcards instead of private key files, and there are apps for smartphone-based SSH authentication, where the private key is protected by the device's hardware.