Can I use a keyed hash of a counter as a secure CSPRNG?
E.g. 4th random block is
If not, what are the possible attacks?
There is a name for "random access PRG" : it's called a pseudorandom function (PRF). Any block cipher is a pseudorandom function (that's more-or-less the definition of what it means to be a secure block cipher), so you can use, say, AES directly. If all you have is a hash function, then the HMAC construction also gives you a PRF.
A possible attack is the length extension attack: you can concatenate anything to the number, if you input the seed before the number. So, for example let's assume you use integers as ASCII strings. Then if you calculate hash("4", seed) = hash(seed || "4"), it is trivial to calculate hash(seed || "4" || "5") = hash(seed || "45") = hash("45", seed).
A possible solution is to have the seed after the number. So, you define hash("4", seed) = hash("4" || seed). Now you can concatenate anything to the seed, but I don't think that will be a problem.
Yes you can. To predict your numbers, an attacker will need to brute-force the key, so it's ok if the key an the hash function are good.
Apart from a hash, you can also use a block cipher in CTR mode. If you have to use a hash, make sure you use HMAC or a length-extension-proof hash function like SHA3 or Blake2, to avoid other people appending things to your numbers.
Also, varoius people suggest that hashes may be secure as CTR mode ciphers and key derivation functions, which both, like your random number generator, use a key and an index to generate a stream of random blocks. Eg: Sha256 as a CTR block cipher and Deriving user keys from ID