Short answer, before someone marks this as a duplicate or answers it with an essay or something:
You're exactly right. This was one of the biggest consequences of the infamous Heartbleed exploit in OpenSSL, which exposed the memory of processes using OpenSSL for TLS to anyone with an Internet connection. It's also significant for cold boot attacks, where someone with physical access to a computer shuts it down and boots a live CD or rips out the RAM and reads it before it loses all charge (moments, normally; much longer with enough refrigerant).
There are mitigations, including storing your keys in a hardware security module (HSM), a physical device designed to be difficult to attack. If your protocol uses forward secrecy -- such as TLS with DHE or ECDHE cipher suites -- new encryption keys are generated for each connection and thrown away relatively quickly (if they're properly erased from RAM, and not counting session caching and resumption mechanisms). In that case, your long-term RSA key would only be used for signatures. Exposing it would allow you to be impersonated and facilitate future MITM attacks, but would not allow decryption of past communications.