Both Salsa20 and ChaCha basically work like this:
- Put the key, the nonce, the sequence number and a constant into a 4x4 matrix of 32-bit words.
- Transform the matrix invertibly with a number of ARX rounds.
- Add the initial matrix into the current state to produce the output block.
Now, as discussed e.g. in this currently featured question, the final addition is required to prevent simple backwards computation from revealing the key. However, my question is: why is the whole initial matrix added in, rather than only the key part?
It seems that the addition of the rest of the matrix is mostly wasted effort, since they are publicly known numbers that an attacker with a known plaintext can simply subtract again.
On the other hand, there is a clear downside of extra memory and computation required. The latter is probably negligible compared to the whole cipher, but the former could be important, especially for hardware implementations. If only the key was added, the nonce and the sequence number would not need to be accessed again, saving a sixth of the state required (if you ignore constants).