On modern CPUs, a fast Cryptographically Secure Pseudo-Random Number Generator runs faster than one cycle per byte. We are talking >40Gbit/s. See numbers there. Top contender are AES-CTR assisted by special instruction, and ARX ciphers like ChaCha.
When using dedicated hardware, the true limit is moving around the generated random bits. We can arbitrarily parallelize CSPRNGs, and we have many efficient designs. For example, Trivium is reported at >120Gbit/s/mm2 using 250nm metal CMOS technology, and state of the art today is about 10nm technology, which is faster and some hundreds time denser.