The RSA problem, which is what you describe, is not known to be equivalent to factoring and there is some evidence both ways. In [BV] it is shown that this barrier ismight be inherent. It is shown in [BV] (using: using a black-box separation technique called metareductions) that any "algebraic" blackmeta-boxreductions straight-line reduction from factoring to the RSA problem can turned into a factoring algorithm, they show that certain restricted class of reductions are not possible. On the other hand, it was shown later in [AM] that in the generic ring model (see [JS]), these problems are equivalent. That is, any speed-up in breaking RSA has to exploit the representation of $\mathbb{Z}_N^*$.
You can read about more related works in §1.3 in [AM].
[BV]: Boneh and Venkateshan, Breaking RSA may not be equivalent to factoring, Eurocrypt'98
[AM]: Aggarwal and Maurer, Breaking RSA Generically Is Equivalent to Factoring, Eurocrypt'09
[JS]: Jager and Schwenk, On the Analysis of Cryptographic Assumptions in theGeneric Ring Model, Asiacrypt'09