It appears that based on your description, the server has a 2048 bit key pair and the clients logging in have 4096 bit key pairs.
In this case, do I have the brute force protection of 2048 or 4096 bits?
For someone to impersonate one of your users, they would have to break the 4096 bit key.
For someone to impersonate the server, they would have to break the 2048 bit key.
My worry is that someone could brute force the private key and login to the server.
Unless you work for some huge financial institution or a government, if someone could break either of those key sizes they are likely not going to go after you. ATM, there is no publicly available way to brute force a 2048 bit key. Someone with the ability to do it could easily make millions of dollars. So I'd say you are safe.