This is a brilliant idea, but I don't know of a way to make it viable.
Presumably, you're thinking you could do something like have a file named "my_photo_1.jpg" where it's been specially constructed so if the first bit of the key is a "1", the encrypted file contains some data revealing that key bit, "my_photo_2.jpg" is structured so that it reveals another key bit, etc. These are called "chosen plaintext attacks".
So that leaves hope for a vulnerability in the encryption algorithm used by the extortionist. Unfortunately for your idea, if the extortionist is using AES, the best chosen plaintext attack published so far is an improved variant of the Square attack, which works on reduced (seven) round variants of AES. Cryptographers study reduced-round variants in order to search for weaknesses in algorithms, but real world applications of AES follow the published standard and use 16 rounds. While the knowledge gained from reduced-round variants is helpful, they can't be applied in practice to solve this problem. Yet.
Chosen plaintext attacks are normally not very practical against a real-world adversary, because most traditional adversaries aren't likely to encrypt a bunch of random messages that might weaken their cyphers. But in this case, where the encryption is hostile and you control all the plaintext, it's a perfect application. This might spur further research into this field.
Unfortunately, this can't work for RSA as you suggest. RSA uses two different keys, a public and a private key, and only the public key is distributed with the malware. (The private key is kept confidential on the extortionist's servers, to be sent to the victim only after the ransom is paid.) RSA is used only to encrypt the random number used as the key to the symmetric algorithm, which is used to encrypt the files. Chosen plaintext encrypted with the public key doesn't reveal anything about the private key.