Let's review the design of ChaCha to see how the nonce, the counter, and te number of rounds all fit into it.
How do we encrypt a sequence of messages $m_1, m_2, \dots, m_\ell$? One way is to pick a sequence of message-length pads $p_1, p_2, \dots, p_\ell$ independently and uniformly at random, and encrypt the $n^{\mathit{th}}$ message $m_n$ with the $n^{\mathit{th}}$ pad $p_n$ as the ciphertext $$c_n = m_n \oplus p_n,$$ where $\oplus$ is xor. If the adversary can guess a pad, you lose; if you ever repeat a pad for two different messages, you lose. Otherwise, this model, called the one-time pad, has a very nice security theorem, but choosing and agreeing on independent uniform random pads message-length $p_n$ is hard.
Can we make do with a short uniform key $k$, say 256 bits long? Approximately, yes: if we had a deterministic function $F_k$ from message sequence numbers $n$ to message-length pads $F_k(n)$ which are hard to distinguish from independent uniform random when $k$ is uniformly distributed, then we could pick $$p_n = F_k(n)$$ and we only need to choose and agree on a 256-bit secret key $k$. We call $F_k$ a pseudorandom function family. This makes our job easier without making it much easier for any adversary even if they could spend humanity's entire energy budget on breaking it.
How do we design our short-input, long-output PRF $F_k(n)$? If we had a short-input, short-output PRF $f_k(n, c)$ which computed a fixed-size block given a message sequence number and an extra input $c$, we could simply generate a lot of blocks for each message, using a block counter for the extra input $c$, and concatenate them: $$F_k(n) = f_k(n, 0) \mathbin\| f_k(n, 1) \mathbin\| f_k(n, 2) \mathbin\| \cdots.$$ How do we design our short-input, short-output function $f_k(n, c)$? If $\pi$ were a uniform random permutation, then the function $S(x) = \pi(x) + x$ would be hard to distinguish from a uniform random function, and almost certainly be noninvertible. We could define $$f_k(n, c) = S(k \mathbin\| n \mathbin\| c \mathbin\| \sigma).$$ Of course, we don't have a uniform random permutation, but if $\delta$ is a permutation without much structure, and if we define $\pi$ by iterating $\delta$ many times, $$\pi(x) = \delta(\delta(\cdots(\delta(x))\cdots)) = \delta^r(x),$$ then $\pi(x)$ will have even less structure than $\delta$—with any luck, so little structure that it will destroy any patterns a cryptanalyst could look for within humanity's energy budget.
Recapitulating, the design of the ChaCha$(2r)$ is as follows:
- Start with a permutation $\delta$ of 512-bit blocks that doesn't have much structure. The permutation $\delta$ is called the ChaCha doubleround. (Why a ‘doubleround’? ChaCha alternates between ‘row rounds’ and ‘column rounds’; $\delta$ does one row round, and one column round.)
- Define the permutation $$\pi(x) = \delta^r(x),$$ the $r$-fold iteration of $\delta$. The number $2r$ is the number of rounds. For instance, in ChaCha20 (the default), we iterate $\delta$ ten times; in ChaCha8 (the smallest unbroken number of rounds), we iterate $\delta$ four times.
- Define the function $$S(x) = \pi(x) + x.$$ This function $S$ is called the ChaCha core.
- Define the short-input, short-output pseudorandom function family $$f_k(n, c) = S(k \mathbin\| n \mathbin\| c \mathbin\| \sigma),$$ where $\sigma$ is a fixed constant with moderate Hamming weight. When unambiguous, $f_k$ is sometimes just called ChaCha, or the ChaCha PRF.
- Define the short-input, long-output pseudorandom function family $$F_k(n) = f_k(n, 0) \mathbin\| f_k(n, 1) \mathbin\| f_k(n, 2) \mathbin\| \cdots.$$ Here we use the $c$ parameter of $f_k$ as a block counter. When unambiguous, $F_k$ is sometimes just called ChaCha, or the ChaCha stream cipher.
- For the $n^{\mathit{th}}$ message, compute the pad $$p_n = F_k(n).$$ Here we use the $n$ parameter of $F_k$ as a nonce.
- Encrypt the $n^{\mathit{th}}$ message $m_n$ by computing the ciphertext $$c_n = m_n \oplus p_n.$$
When you are using ChaCha, as in the NaCl crypto_stream_chacha_xor(output, msg, len, n, k)
, your obligations are to choose $k$ uniformly at random and never reuse the nonce $n$ with the same key $k$. The counter is an implementation detail that does not concern you in most protocols.
Note 1: You almost certainly shouldn't use ChaCha directly either; you should use an authenticated cipher like ChaCha/Poly1305 or NaCl crypto_secretbox_xsalsa20poly1305
. Unauthenticated data is pure evil—don't touch it!
Note 2: That ChaCha's counter enables random access to blocks within a message also shouldn't concern you; your messages should be short enough that a forgery won't waste much memory before you are guaranteed to realize it's a forgery and drop it on the floor. Use the nonce for random access to a sequence of authenticated messages instead so you're not tempted to reach inside a box of pure evil.
To address the specific questions you asked:
Does the counter at position of 13th byte actually increment by one? Can I extract the number of iterations from the state of ChaCha20?
The counter increments for each block within a single message, as illustrated above.
The number of iterations (or ‘rounds’) is not encoded into the state. The number of iterations for ChaCha20 is always 20. If you have ciphertexts under ChaCha12 and ChaCha20 with an unknown key, you can't tell whether they were made with ChaCha12 or ChaCha20 either.
In particular, the ChaCha20 core, $\operatorname{ChaCha20}_{\mathit{key}}(\mathit{nonce}, \mathit{counter})$ permutes the 512-bit state $(\mathit{key}, \mathit{nonce}, \mathit{counter}, \mathit{constant})$ (encoded in some bit order) with 20 rounds to produce a single 512-bit block of pad at a time; the ChaCha20 cipher then moves on to using $\operatorname{ChaCha20}_{\mathit{key}}(\mathit{nonce}, \mathit{counter} + 1)$ for the next block, and then $\mathit{counter} + 2$, and so on.
From the specification, I'd say that the state gets randomized after as much as one iteration.
There's an illustration of the diffusion of a change in a single byte of the Salsa20 core here: https://cr.yp.to/snuffle/diffusion.html (Salsa20 is closely related to ChaCha; they have almost the same security.)
Does this mean then that nonce can also be made public (just like IV for block ciphers) without compromising the security? (of course provided that key stays confidential)
Yes. Not only can it be public, but it can be predictable in advance—unlike a CBC IV.
The security contract for ChaCha20 obliges you never to repeat a nonce with the same key, and obliges you to limit the messages to at most $2^\ell\cdot 512$ bits long, where $\ell$ is the number of bits reserved for the counter—in NaCl, $\ell = 64$ so messages can be of essentially arbitrary length, while in RFC 7539 as used in, e.g., TLS, $\ell = 32$ so messages are limited to 256 GB which is more than enough for sensible applications which break messages into bite-sized pieces to be authenticated anyway—you are using this as a part of the authenticated cipher ChaCha/Poly1305 or similar, right?
Neither the nonce nor the counter need be secret in the security contract; normally they are prescribed by the protocol and algorithm, e.g. to be a message sequence number starting at 0, and a block sequence number starting at 0, respectively.
It is still unclear to me what the function of counter is. Why not just use larger 128bit nonce, instead of a 32bit counter + 96bit nonce?
If you used a 128-bit nonce, your messages would be limited to 32 bytes long.