On modern CPUs, a fast Cryptographically Secure Pseudo-Random Number Generator runs sizably faster than one cycle per byte. We are talking >40Gbit/s. See numbers [there][1]. Top contenders are [AES][2]-[CTR][3] assisted by special instruction, and [ARX][4] ciphers like [ChaCha][5].

When using dedicated hardware, the true limit is moving around the generated random bits. We can arbitrarily parallelize CSPRNGs, and we have many efficient designs. For example, [Trivium][6] is [reported][7] at >120Gbit/s/mm<sup>2</sup> using 250nm metal CMOS technology, and state of the art today is about 10nm technology, which is faster and some hundreds time denser.

Update per [comment][8]: All modern CSPRNGs require $O(n)$ bit operations to produce $n$ bits, and we want [little-o][9] notation for an interesting figure. [ChaCha][5] with $r$ rounds (see [source][10]) produces $16$ words each 32-bit with computational cost dominated by $16(r+1)$ 32-bit additions and $16r$ 32-bit XORs. Using NAND gates, a [ripple-carry adder][11] is 9 gates, XOR is 4 gates, thus the cost is $13r+9$ NAND bit operations per bit produced. For Chacha8 (which is hoped to have 128-bit security), the cost is like $o(113n)$ NAND bit operations to produce $n$ bits (with no consideration for circuit depth).


  [1]: https://bench.cr.yp.to/results-stream.html
  [2]: https://en.wikipedia.org/wiki/Advanced_Encryption_Standard
  [3]: https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#CTR
  [4]: https://en.wikipedia.org/wiki/Block_cipher#ARX_(add%E2%80%93rotate%E2%80%93xor)
  [5]: https://cr.yp.to/chacha.html
  [6]: https://en.wikipedia.org/wiki/Trivium_(cipher)
  [7]: http://www.ecrypt.eu.org/stream/e2-trivium.html
  [8]: https://crypto.stackexchange.com/posts/comments/138736
  [9]: https://en.wikipedia.org/wiki/Big_O_notation#Little-o_notation
  [10]: https://crypto.stackexchange.com/q/26437/555
  [11]: https://en.wikipedia.org/wiki/Adder_(electronics)#Ripple-carry_adder