The standard criterion for security is indistinguishability under adaptive chosen-ciphertext attack, or IND-CCA2. What this means is that the adversary is given:
- the public key $(n, e_B)$, and
- an oracle that answers queries of the form: What is the plaintext for the ciphertext message $c$?
The adversary's task is to find any pair of messages with a pattern they can distinguish in the corresponding ciphertexts. Specifically, perhaps after interacting with the oracle, the adversary chooses two messages $m_0 \ne m_1$, and asks to be challenged with the ciphertext $c_b$ for $m_b$ where $b$ is a secret coin toss not known to the adversary; the adversary then wins the game if they guess correctly what $b$ was.
If ‘encryption’ is $m \mapsto m^{e_B} \bmod n$, then this is very easy! The adversary can furnish any pair of plaintexts $m_0 \ne m_1$, and check whether $c_b \stackrel?= {m_0}^{e_B} \bmod n$ or $c_b \stackrel?= {m_1}^{e_B} \bmod n$ to determine what $b$ was. What this illustrates is that public-key encryption must be randomized so that the adversary cannot simply confirm guesses about what the plaintext is.
‘But,’ you object, ‘I said the plaintext is a random key which the adversary cannot predict!’ Well, security in your scenario is weaker than security in the IND-CCA2 scenario, because you've added extra assumptions about how the legitimate users use the cryptosystem. But OK, let's say you add that assumption.
For example, let's say the legitimate users use RSA-2048 to encrypt AES-256 keys for AES-GCM, and let's say they pick the exponent that gives the best performance: $e_B = 3$. Now when you send me $$c = {k_1}^{e_B} \bmod n = {k_1}^3 \bmod n,$$ I can simply compute the real number cube root $\sqrt[3] c$ to recover what $k_1$ was, because as an integer, $0 \leq k_1 < 2^{256}$, so that ${k_1}^3 < 2^{768} \lll n$, which means the $\bmod n$ part never kicked in with parameters of this size! Oops.
You might object that $e = 3$ is bad, but even with larger exponents like $e = 65537$ there are all manner of elaborate attacks using black magic like continued fractions or lattice algorithms on structured messages such as 256-bit strings. The problem is that the RSA trapdoor permutation $x \mapsto x^e \bmod n$ is bad at concealing structured messages. It's only good at concealing uniform random elements of $\mathbb Z/n\mathbb Z$.
What's different about RSA-KEM is that you choose a secret integer $x$ uniformly at random below $n$ independently for each message, and then hash it to derive your AES-256 key $k_1 = H(x)$ while transmitting the encapsulation $y = x^e \bmod n$ so the recipient can recover $x$. Note that $x$ is not restricted to $0 \leq x < 2^{256}$; it is only restricted to $0 \leq x < n$. This means that any adversary defined in terms of a generic hash function $H$ can easily be shown to be able to compute arbitrary $e^{\mathit{th}}$ roots modulo $n$, meaning that such an adversary is guaranteed to be able to solve the RSA problem.
(And the most efficient exponent 3 is just fine with RSA-KEM, as it is with any serious public-key encryption scheme built out of RSA.)