...random enough..
Let's just focus on that as it forms the nub of your question. Yes,$$\text{random (pseudorandom) bitstream} \oplus \text{secret message}$$ is very simple, works and is in common usage. But the first term in this encryption function masks great (and necessary) complexity. In order to be secure, the bitstream must comprise independent and uniformly distributed numbers, typically with bias not exceeding $2^{-64}$. How do you obtain them? Therein lies your complexity.
You can make them with a physical device. If we exclude zany methods like dice and aquarium fish, we're left with some type of electromagnetic apparatus. These days, that will take the form of either a laser or diode. Both create the original entropy for distillation into random numbers based on quantum indeterminacy. Quantum mechanics are pretty complex.
Or you can expand a little entropy like say a password, into a long stream of pseudo random numbers with a CSPRNG (and perhaps a key derivation function). The need for non invertability/non predictability of the output and to guarantee acceptable bias, requires complexity. And so cryptographic primitives are complex. If it wasn't complex, you could just roll it backwards.
Therefore, what you've really asked is:- $$\text{complex to make random (pseudorandom) bitstream} \oplus \text{secret message}$$
You've simply moved the complexity upstream of the XOR operator.