The CTR mode of encryption is defined in general for any cryptographically strong pseudo-random function (PRF). You can build such a PRF from a hash function.
For CTR, you produce a key stream by concatenating:
$$F(k,0) || F(k,1) || ... || F(k,m)$$
where $F$ is your secure PRF, $k$ is your key, and $m$ is the the length of your plaintext divided by the output size of $F$. To rewrite it in your mode's notation:
b[n] = F(k,n)
But $F$ can be any secure PRF. It can be AES but it doesn't have to be, AES is just the popular choice.
Since you want to use SHA-256, you can use a hash-based PRF. You have effectively tried to build one, but a commonly accepted PRF, like HMAC, would be better. If you use HMAC with SHA-256 as your hash function, you have effectively built CTR mode using SHA-256. To do so, your b[n]
would change from your function:
b[n] = SHA-256(SHA-256(nonce XOR key) XOR n)
to the HMAC function:
b[n] = SHA-256(key XOR opad || SHA-256(key XOR IPAD || (nonce || n)))
The HMAC scheme will require two hashes every block, whereas your scheme allowed you to only calculate the first hash once, so the HMAC scheme will be roughly twice as slow.
It's worth noting that HMAC is not proven to be a perfect PRF, but it's widely regraded as a PRF. It's definitely a safer PRF choice than a home-made construction. I can't speak as to how secure your scheme is -- it certainly looks plausible, but that doesn't really mean anything -- but it's none better than HMAC.
And since you're so close to HMAC already, if you need to use SHA-256 to build CTR-style encryption, I would recommend you use that (or some other hash-based PRF) instead.
Also, on a side note regarding your closing comment: CTR produces a stream cipher. We commonly use a block cipher to build it, but CTR mode is actually a stream cipher. So we can build a stream cipher from SHA-256, but that's distinct from being able to build a block cipher from it. I think what you meant is that you have a set of schemes you're thinking about (OFB, CFB, CTR, and maybe others) and want to know which of them a hash can be used to fulfill. Given the ability to turn a hash into a PRF, the answer would include CTR.
for i,b in blocks: b[i] = xor(b[i], Sha256(key+iv+i))
seems like such an obvious cipher. (key+iv) means that every message has a unique starting point in the "OTP". iv and i are public. $\endgroup$