Here is an alternate, easier way.
Fix the notation. Let $N = pq$ for two large primes $p$ and $q$. Alice 's public key is $N$ while her private key is $\lambda = \operatorname{lcm}(p-1,q-1)$.
The message space is $\mathcal{M} = \mathbb{Z}_N$. The encryption of a message $m \in \mathcal{M}$ is given by $C = (1+mN)r^N \bmod N^2$ for some random integer $r \in [1, N)$.
To get the decryption of a ciphertext $C$ in a verifiable way, Bob engages in the following protocol with Alice:
- Bob computes $R = C \bmod N$ and sends $R$ to Alice;
- Alice using her private key computes $r' = R^{N^{-1} \bmod \lambda} \bmod N$ and sends $r'$ to Bob;
- Bob checks that $(r')^N \equiv R \pmod N$. If so, Bob recovers plaintext $m$ as $m= \frac{C\cdot (r')^{-N} -1 \bmod N^2}{N}$. Otherwise, Alice
cheated.
This presents the further advantage that plaintext $m$ is unknown to Alice. Moreover, the bandwidth requirements are minimal: Alice and Bob only exchange $2\log_2 N$ bits ($R$ and $r'$).
Computations can be slightly sped up. Here is a variant. Alice defines her private key as $d = -N^{-1} \bmod \lambda$.
- Bob computes $R = C \bmod N$ and sends $R$ to Alice;
- Alice using her private key computes $r'' = R^d\bmod N$ and sends $r''$ to Bob;
- Bob computes $S = (r'')^N \bmod N^2$ and checks that $S \cdot R\equiv 1 \pmod N$. If so, Bob recovers plaintext $m$ as $m= \frac{C\cdot S -1 \bmod N^2}{N}$. Otherwise, Alice
cheated.
The correctness follows from the next proposition.
Proposition. For any $r \in \mathbb{Z}_{N^2}$, one has $r^N \equiv (r\bmod N)^N \pmod {N^2}$.
Proof. Write $r = r_0+ r_1N$ where $r_0 = r \bmod N$ and $r_1 = \lfloor r/N\rfloor$. The binomial formula yields $$r^N=(r_0 + r_1N)^N \begin{array}[t]{l}= \displaystyle\sum_{k=0}^N {N \choose k} r_0^{N-k} (r_1N)^k\\ = r_0^N + Nr_0^{N-1}(r_1N) + N(N-1)r_0^{N-2}(r_1N)^2 + \dots\end{array}$$
Hence, reducing modulo $N^2$, we obtain $r^N \equiv r_0^N \equiv (r\bmod N)^N\pmod {N^2}$.