Can Shor's algorithm compromise RSA when both the public and private key are secret?

Question

If RSA is to create a public-private key pair and encryption is performed on plain test P to create ciphertext C, given P and C could Shor's algorithm be used to find either of the public and private keys? You have both the plain text and cipher text but neither of the keys.

The fact that it seems pointless to use RSA if both the public and private keys are secret has come up a few times now so let me explain. I would like to use RSA because it has the property that if plain text P is encrypted with key K1 and the result is encrypted with key K2, assuming K1 and K2 have the same modulus base, this will produce the same ciphertext as encrypting P with K2 then K1.

If there is another more secure encryption scheme that has this property then I would be very interested in knowing about it. Otherwise, I will need to use RSA which is why I am asking about RSA's strength against quantum computers.

Also, the module base would be publicly known as well. The only secret information would be the exponents in the key pair.

Purpose

It has been asked what the value of key commutativity is. It allows one party to encrypt a secret text P (known only to party A) with the secret key K of another part (known only to party B). This is how:

let P = plain text (known only to party Alice)
let C = ciphertext (known only to party Alice)
let K = key (known only to party Bob)
let U = intermediate key (known only to party Alice)

Alice encrypts P with U and sends it to Bob. Bob encrypts this with K and sends this back to Alice. A removes the initial encryption (using U's decrypting pair) getting C.

C is the same as if P were encrypted with K without the involvement of the intermediate key because the keys commute.

Schematic:

Alice and Bob can only see things that are on their "side" (both the keys and the texts). the wall of equal signs represents this separation. the arrows represent text being transferred from one place to another and the keys intersecting the arrows represent encryption of the text traveling through them. I hope this is a clear explanation of this process. I'm not familiar with any traditional way of drawing these sorts of schematics. I someone could direct me to some information on how to draw one I would be happy to learn it and rewrite my schematic in that format.

Answer 1

Actually, if RSA is being used in a deterministic way (and the public exponent $e$ is relatively small), someone could recover the value $N$.

We know that $P^e = C \bmod N$; that's equivalent to $P^e - C = kN$ for some integer $k$; if $e$ is small, then Shor's algorithm might be able to factor $P^e - C$; allowing you to recover $N$. Alternatively, if you have two plaintexts/ciphertexts from the same key, then all you need to do is $\gcd(P_1^e - C_1, P_2^e - C_2)$, which can be done on a conventional computer.

Now, for public key encryption, we never use determanistic encryption. However, for generating signatures, we sometimes do; this might actually be applicable there.

I would also second Maarten's comment that if you really intend to keep the public key secret, then there are symmetric algorithms that are far more practical (not to mention Quantum-Resistant).