Here's a quote from Douglas Stinson:
“[i]f a cryptosystem can be ‘broken’ in some specific way, then it would be possible to efficiently solve some well-studied problem that is thought to be difficult. For example, it may be possible to prove a statement of the type “a given cryptosystem is secure if a given integer n cannot be factored.” [...] [B]ut it must be understood that this approach only provides a proof of security relative to some other problem, not an absolute proof of security. This is a similar situation to proving that a problem is NP-complete [...].”
Source: Stinson, D. R. (2006). Cryptography, Theory and Practice. Chapman and Hall, CRC, 3rd edition. Chapter 2, section 2.1, page 45.
Does this apply to RSA? Suppose I find the private exponent by some other way other than factoring $n$. Then I would have broken RSA without factoring.
So although Stinson's example mentions factoring, we can't immediately think of RSA here because there's no proof that factoring is the only way to recover a private key. What do you say?