What's the size of a 4096 bit private key?
It depends.
The range goes from 512 bytes (only $d$, no formatting, no encoding) up to 5632 (full parameter set, maximal $e$ size, no formatting, hexadecimal encoding) bytes for two-prime RSA. If you also consider multi-prime RSA, the range is much wider.
So what influences the size of the stored private key then?
- How do you encode the stored data?
- No encoding, this will give no increase in storage need
- Base64 encoding, this encodes 3 bytes of data into 4 bytes of encoded data
- Hexadecimal encoding, this will encode 1 byte of data into 2 bytes of encoded data
- How do yo format your data?
- No formatting, this also means no overhead
- ASN.1, this will give some overhead to make the structures parseable and the overhead will increase if you use XER instead of DER, the relevant standard is PKCS#1 (PDF) here as well as PKCS#8 for containers and PKCS#12 (PDF via Archive.org) as additional wrapper.
- PEM, this will give you some overhead for internal structure and the header and footer of the data block
- How much data are you storing?
- The absolute minimum $p$, which will allow you to factor $n$ using simple division and will allow you to later derive $d$ "the usual way". This approach provides optimal storage requriments, assuming you're not willing to run the algorithms from 3. which would reduce the need for stored bits of $p$.
- Only the private exponent $d$, which will give you a good storage characteristic, but also requires external supply of the public modulus $n$ and the public exponent $e$ for proper operation
- A complete minimalistic approach $(n,p',e)$, this will require you to store $e$, but you can optimize it by hard-coding it as $F_4$. The technique is to store only a minimalistic amount of information of $p$, but enough to recover $p,q$ as per "On Factoring Arbitrary Integers with Known Bits" by Herrman and May (PDF) and "Small Solutions to Polynomial Equations, and Low Exponent RSA Vulnerabilities" by Coppersmith (PDF) . This reduces the storage requirement to 5120 bits, however I do no recommend this as it is highly non-standard and will give you large interoperability problems for a marginal storage saving.
- The smallest (trivial) complete set $(p,q,e)$ which will allow you to recover $n$ using standard multiplication and will allow you to recover $d$ the usual way.
- A middle-ground minimum $(n,d)$, this will not allow you to factor $n$ without external supply of $e$ (which may be an agreed-upon constant), but is enough to decrypt messages "classically"
- Only the bare minimum $(n,e,d)$, this will allow you factor $n$ somewhat quickly if needed and it will allow you to do all operations without external data supply
- A middle ground amount of data $(n,e,d,p,q)$, this will take away the "heavy" step of factoring $n$ but still leaves some computations to be done if you want that speed-up from the CRT.
- The full set $(n,e,d,p,q,d_p,d_q,q_{\text{inv}})$, this allows you to load the data and outright start with the computations, including the CRT speed-up
How does it compare to the public key?
Look at the above list of influencial factors and adapt the third major point, a public key "only" has the option $(n,e)$ there, meaning it can be larger than the private key or it can be significantly smaller, if you include the associated data in the private key.