Assume we have the following setup:
- A client with trusted storage and computing capabilities (e.g. a smartcard)
- A server with trusted computing and short-term storage capabilities (e.g. RAM + CPU, possibly with something like Intel SGX). The server has no trusted large-scale long-term storage capabilities and may only store small amounts of data confidential and integrity protected (like the HTTPS private key).
The problem is: The server should be able to be shut-down and started-up, no passwords should be involved and the server has no HSM, yet the server should be able to provide somewhat secure access to some data without the clients needing to decrypt it themselves (for complexity reasons). So the storage need to be encrypted and the transfer (-> TLS) as well.
The solution is now (what I call it): blinded decryption.
The server uses some homomorphic encryption scheme (e.g. EC-ElGamal or RSA) with the message space $\mathcal M$. He chooses a random $k\in \mathcal M$ and uses $H(k)$ ($H:\mathcal M \rightarrow \{0,1\}^{256}$) as the key for the authenticated encryption of the data. The server now either stores the (asymmetric) encryption of $k$, called $\mathcal E(k)$ in his trusted area of the drive(s) or may store it in an untrusted section (with back-ups) if server authentication is required and the private key for this authentication is already stored in the trusted area.
For the temporary unlock of the encrypted data, the server loads $\mathcal E(k)$. Then he blinds it using some operation $f(\cdot,\cdot)$ (multiplication for ElGamal and RSA, addition for EC-ElGamal) using some random $r\in \mathcal M$ as $c=f(\mathcal E(k),\mathcal E(r))=\mathcal E(g(k,r))$ with $g(\cdot,\cdot)$ being the "inner homomorphism" (same as $f$ in many cases). The $r$ is kept available in trusted short-term memory and the $c$ is sent to the client.
The client decrypts $c$ using his trusted device and returns $c'=g(k,r)$ to the server. Finally the server unblinds $c'$ using his $r$ and uses the obtained $k$ to derive $H(k)$ and allow access to the data.
Now (finally) the question:
Given the above and standard assumptions (RSA-assumption, DDH-assumption in ECC and $\mathbb Z_p^*$,$H$ is a random oracle, the symmetric encryption is secure and authenticated,...) is it safe to instantiate $\mathcal E$ with textbook RSA?
As pointed out in the comments, every good question about "is this secure?" requires a threat model, so here's mine:
The security of the whole protocol is broken if an attacker is able to learn the secret symmetric key $H(k)$ while it's valid. The attacker may not compromise the server (i.e. he can't control/spy on RAM / CPU and may not learn the stored $\mathcal E(k)$). An attacker not breaking into the server may have successfully attacked the client (except for the trusted device) and he may be able to completely modify and read the network traffic. I think an attacker without having broken into the server may be computationally unbounded.
If not clear until now, the instantiation of $\mathcal E(k)$ is $\mathcal E(k):=k^e \bmod N$ with $e,N$ being standard RSA parameters.