Quick summary.
Ransomware works by a public-key key encapsulation mechanism: pick a secret key $h$ and an encapsulation $\sigma$ of it for the malware operator's public key, so that only the malware operator can recover $h$ from $\sigma$ using their private key; then encrypt all the files using a symmetric cipher under the key $h$, erase $h$, and pay the malware operator ransom to get $h$ back from $\sigma$.
It's easy to build a public-key key encapsulation mechanism out of a public-key key agreement scheme such as ECDH with a long-term public key for the recipient: Generate a single-use anonymous key pair for the sender, combine the sender's private key with the recipient's public key to derive $h$, erase the sender's private key, and transmit the sender's public key as the encapsulation of $h$. Using that as a subroutine, the malware then works just like it would with any public-key key encapsulation mechanism.
Details.
First, for reference, here's a way malware on the host can use a public RSA modulus $n$, say between $2^{2047}$ and $2^{2048}$, with secret factorization $n = pq$ known only to the malware operator.
- Pick a nonnegative integer $x$ below $n$ uniformly at random using the host's random number generator, e.g. /dev/urandom on Unix or CryptGenRandom on Windows.
- Compute and store $y = x^3 \bmod n$.
- Compute $h = H(x)$, where $H\colon \mathbb Z/n\mathbb Z \to \{0,1\}^{256}$ is a random oracle. E.g., you might instantiate $H$ with SHA-256 on the little-endian $\lceil\log_{256} n\rceil$-byte encoding of the least nonnegative residue of $x$.
- Compute and store $c = \operatorname{AES-GCM}_h(\mathit{data})$, where $\mathit{data}$ is whatever important data are on the host's persistent storage that you're ransoming.
- Erase $x$, $h$, and $\mathit{data}$.
- When the hapless user pays the ransom, transmit $y$ and a proof of payment to the malware operator.
On receipt of $y$ and proof of payment, the malware operator can use secret knowledge of $p$ and $q$ to compute $x = y^d \bmod n$ where $d$ solves $3d \equiv 1 \pmod{\operatorname{lcm}(p - 1, q - 1)}$, and from there derive $h = H(x)$ for the luser to use to decrypt $c$ with AES-GCM.
The security of this scheme relies on the difficulty of computing cube roots of uniform random integers modulo $n$ without knowledge of the factorization of $n$. There are some integers whose cube roots modulo $n$ are very easy to compute: the ones that are perfect cubes as integers in the first place, so that ordinary real number cube root algorithms can compute them. But the vast majority of integers are not perfect cubes, so the probability that $y$ happens to be a perfect cube is negligible, and the cheapest algorithms we know to compute $x$ or $h$ given $y$ with nonnegligible probability work by factoring $n$, namely ECM and GNFS.
How does it work with ECDH?
Fix a standard public field $k$, say $\operatorname{GF}(2^{255} - 19)$. Fix a standard public curve $E/k$, say $y^2 = x^3 + 486662 x^2 + x$. Fix a standard public $k$-rational point $G \in E(k)$ of large prime order, say $G = \pm x^{-1}(9)$. Fix a public $k$-rational point $P \in E(k)$ for which the malware operator knows a secret integer $s$ such that $$P = [s]G = \underbrace{G + G + \dots + G}_{\text{$s$ times}}.$$ Using these parameters, the malware on the host will:
- Pick a nonnegative integer $t$ between $2^{254}$ and $2^{255}$ that is a multiple of 8, uniformly at random using the host's random number generator, e.g. /dev/urandom on Unix or CryptGenRandom on Windows.
- Compute and store $Q = [t]G$.
- Compute $h = H([t]P)$.
- Compute and store $c = \operatorname{AES-GCM}_h(\mathit{data})$.
- Erase $t$, $h$, and $\mathit{data}$.
- When the hapless user pays the ransom, transmit $Q$ and a proof of payment to the malware operator.
On receipt of $Q$ and proof of payment, the malware operator can use secret knowledge of $s$ to compute $$[s]Q = [s\cdot t]G = [t\cdot s]G = [t]([s]G) = [t]P,$$ and from there derive $h = H([s]Q) = H([t]P)$ for the user to use to decrypt $c$ with AES-GCM.
This scheme is sometimes called ECIES. The pair $(s, P)$ serves as the recipient's long-term key pair, while the pair $(t, Q)$ serves as the sender's single-use key pair, for public-key key agreement with an elliptic-curve Diffie–Hellman function.* Choosing a single-use anonymous key pair like this turns a public-key key agreement scheme into a public-key key encapsulation mechanism.
The security of this scheme relies on the difficulty of computing discrete logarithms in the group $E(k)$. The parameters are all public; all that the adversary—in this case, the hapless user who has been afflicted by ransomware, or the IT specialists they hire to recover their data without just paying the ransom—doesn't know are the secret integers $s$ and $t$, the AES-GCM key $h$, and the poor user’s data.
* The curve chosen, Curve25519, admits a fast algorithm to compute $x([s]G)$ given only $s$ and $x(G)$, which is why the Diffie–Hellman function is called X25519; we actually don't ever compute the $y$ coordinates or full points, just the $x$ coordinates above, but the notation got cumbersome when everything was written as $x(Q)$, $H(x([t]P))$, etc.
