We are going to integrate some hardware accelerator for RSA into our next system on chip. This chip can accelerate the core RSA operation, i.e. modular exponentiation $y=A^x \mod(N)$. The private key ($x$) to use can either be provided from the outside or will be stored inside the hardware core, unaccessible to any other hardware /software component. Side-channel attacks are not an issue in this question.
We assume that the software can query the accelerator to do the modular exponentiation using the on-chip key ($x$) and just gets the result ($y$) for any arbitrary $A$ he provides. Would this setup be susceptible to any oracle-kind attack which would allow extraction of the on-chip key $x$? If so is there any countermeasure against this (using padding according to PKCS #v1.5 or RSASSA-PSS)?