First note that the key size of RSA is determined by the size of the modulus $N$ denoted $n$. As the modulus is always the same size for both the public key and private key. So this question is probably about the size of the exponent rather than the actual key size.
The private key exponent must be large otherwise RSA becomes vulnerable. This is the attack by Dan Boneh and Glenn Durfee
, as fgrieu already explained in his answer. Of course if the private exponent is sufficiently small it could also be brute forced by an attacker.
When setting the public key exponent to a small value, such as the fourth prime of Fermat ($2^{16} + 1$ or $65537$) and using the standard key pair generation procedures the private key will with a very high probability be large enough, given a high enough key size.
Now the modulus and public exponent are considered public knowledge. So there is no good reason to choose a high public exponent to avoid brute force. The public exponent simply needs to be safe within the RSA cryptosystem. Furthermore, a small RSA key, especially one with as few bits as possible, is most efficient when calculating $p^e \bmod N$ (where $p$ is the padded message).
So, in conclusion, no, you should not use a large public exponent. The public key size is always the same size as the private key size, so that part of the original question doesn't make sense.