What is the cryptographic strength of sha512ctr when I simply concatenate a counter and 64 bytes from a cryptographic random number generator (in this case RNGCryptoServiceProvider from .net), and some additional optional data such as the user environment block?
Your proposal is more or less what NIST SP 800-90A describes as "Hash_DRBG". From a pure cryptographic point of view, there's nothing bad with it. A few remarks can be made from a more engineering standpoint:
It's not very fast. SHA-512 is a reasonably fast hash function on 64-bit architectures, but that statement is about the amount of input data that SHA-512 can absorb. Each SHA-512 instance processes input data as blocks of 128 bytes, but outputs only 64 bytes. One can hope for, possibly, about 150 MB/s of output on a recent x86 CPU. By comparison, AES/CTR (with the dedicated opcodes) will go at 5 GB/s. Moreover, on 32-bit architectures, especially small ones (microcontrollers), SHA-512 is terrible. AES/CTR (on systems with a hardware implementation of AES) and ChaCha20 (on all others) will be a better bargain.
If you concatenate a counter, a 64-byte seed, and "additional data", and the total length exceeds 111 bytes, then each SHA-512 instance will require the processing of several blocks (111, not 128, because of the SHA-512 internal padding), and performance will further decrease. It would be better to first process the seed and the additional data into an aggregate 64-byte value (e.g. with a SHA-512 call), and use that aggregate value along with the counter into SHA-512/CTR.
"Userland" PRNG have some potential security issues. For instance, a call to
fork()(on a Unix-like system) will duplicate the internal state; the parent and the child will then produce the same pseudorandom output from their shared state. A similar point can be made about virtual machine snapshots. Generous use of an uncloneable random source can help mitigate such issues (e.g. RDRAND on recent x86 CPU). Going through the kernel (
/dev/urandomor equivalent) can help with that: the kernel is resilient to cloning by
fork(), and it has direct access to the hardware, allowing to use random sources such as RDRAND, which are tricky to invoke from .NET code.
Assuming an attack that can compromise the internal state, the attacker will learn enough to not only predict the next pseudorandom bytes, but also previously generated output, by "rewinding" the counter. If you do not use fresh randomness, a compromise of future output, under that scenario, is unavoidable; however, forward secrecy should be maintained: even after full memory compromise, past generator output should be forever out of reach. Such a property can be obtained by "ratcheting" (in simple words, replacing the current seed with a one-way hash of it). If you look at NIST SP 800-90A, this is what happens at the end of the Hash_DRBG_Generate process, after producing the requested output.