While encrypting AES in CTR mode, the input to the AES encryption function is the either a combination of IV + $m$ bit counter or is either $0$ IV bits with an $m$ bit counter. It is however rare to see the full usage of the input block ($128$ bits) as just a counter. Since CTR mode essentially converts a block cipher to a stream cipher, the implementer must be careful that the XOR of ciphertexts should not reveal XOR of plaintexts. For this requirement, the $(nonce, key)$ pair should be unique for each invocation. I'm particularly interested in file encryption, or a large (in size) session of encrypted communication over a network , (say file download > 64 GB).
My encryption scheme: Use a 128 bit IV to populate the initial block. For every requirement of a 16 byte keystream increment the counter and add it with the initial block populated with the IV. (EDITED)
This will be the protocol: $C_{i} = AES_{k}(IV + i) \oplus P_{i}$ where $i$ is 64 bit counter, $IV$ is $128$ bit nonce and $+$ is arithmetic addition modulo $2^{128} -1$.
- Is this secure (confidential sense)?
- Will there be collisions in the input block to the AES function since we have already populated the initial block, with random IV (Can modular addition of counter cause collision) ?
- Is this a standard way of doing AES CTR ?
- What is the proper way to check for overflow? Should I check whether initial block (treated as a big endian integer) equals to $2^{128} -1$ or only whether $i$ equals $2^{64} -1$ and then raise an exception?