If we want to compact an _existing_ RSA private key expressed as $(N,e,d,p,q,d_p,d_q,q_\text{inv})$, we can reduce it to $(e,p,q)$ and easily recompute the rest as:

$\begin{align}
N&=p\cdot q\\
d&=e^{-1}\bmod\operatorname{lcm}(p-1,q-1)\;\text{ or }\;d=e^{-1}\bmod((p-1)\cdot(q-1))\\
d_p&=d\bmod(p-1)\;\text{ or equivalently }\;d_p=e^{-1}\bmod(p-1)\\
d_q&=d\bmod(q-1)\;\text{ or equivalently }\;d_q=e^{-1}\bmod(q-1)\\
q_\text{inv}&=q^{-1}\bmod p
\end{align}$

It is possible to gain a few more bits; for example the low order bits of $p$, $q$ and $e$ are known to be set and need not be stored; further we know that $p\bmod6$ is either $1$ or $5$, thus it is enough to store $\lfloor p/6\rfloor$ and an extra bit, etc.. All in all, any RSA private key with $k$-bit public modulus $N$ and common (small) $e$ can be stored in about $k$ bits.

---
If we want a compact representation of private keys that we are free to choose, we can fix $e$ (removing need to store it) and decide to generate keys using some well defined deterministic procedure employing some Cryptographically Secure Pseudo Random Number Generator, and store the seeds of that CSPRNG, rather than the private keys. Whenever we need a private key, we (re)generate it from its seed. That has a performance issue, with workaround, see [kasperd's answer][1].

We can safely use a (truly random) 160-bit seed: enumerating the possible seeds, generating the corresponding key, and testing if it matches one of the public keys, will remain virtually impossible, and more costly than factoring (for common RSA keys sizes, and billions public keys). We can even use a passphrase, salt (user identifier), and a password-based key generation function, see [this answer][2].

  [1]: http://crypto.stackexchange.com/a/30228/555
  [2]: http://crypto.stackexchange.com/a/24517/555