3
$\begingroup$

Assuming that I want to encrypt a 1 GB file with e.g. AES in GCM mode or ChaCha20Poly1305.

[I'm specifically referring to the cryptography module for Python: https://cryptography.io/en/latest/hazmat/primitives/aead/ but I can't find anything in the documentation]

For "non-AEAD" ciphers, the syntax is basically

cipher = Cipher(key)
cipher.encrypt(data)

and I can call .encrypt() as many times as I like.

However for AEAD ciphers, the syntax is

aead = AEAD(key)
aead.encrypt(nonce, data, associated_data)

So I need to pass "nonce" and "associated_data" as arguments when I want to encrypt "data".

I understand that the last 16 bytes of the cipher text are used to ensure the integrity and that the AEAD ciphers can operate on data up to 2^31-1 bytes.

Does that mean that I can't read and encrypt the file in chunks, I have to load the whole file into memory and there's essentially no way to encrypt anything beyond 2 GB? (It can't possibly be correct to pass the nonce as an argument every time I encrypt a data chunk...)

$\endgroup$

1 Answer 1

4
$\begingroup$

This is a good question that should be answered more frequently in documentation.

Does that mean that I can't read and encrypt the file in chunks, I have to load the whole file into memory and there's essentially no way to encrypt anything beyond 2 GB?

Nope, you can and should encrypt files in chunks. The main benefit aside from reduced memory usage is that you detect modification/corruption sooner.

I would recommend 16, 32, or 64 KiB as the chunk size and that you use a unique key for each file.

(It can't possibly be correct to pass the nonce as an argument every time I encrypt a data chunk...)

Actually, it is. However, the nonce should change each time.

  • With AEADs that have a short nonce (e.g. 96 bits) like AES-GCM and ChaCha20-Poly1305, it's best to use a counter starting at 0 and increment it for each chunk.
  • With AEADs that have a longer nonce (e.g. 128, 192, or 256 bits) like XChaCha20-Poly1305 and AEGIS-256, it makes more sense to randomly generate the nonce for each chunk. Otherwise, you may as well just use an AEAD with a shorter nonce.

Unfortunately, it's not this simple.

  1. You need to prevent an attacker rearranging/duplicating chunks. A counter nonce solves this problem. With random nonces, one approach is to use the previous authentication tag as associated data for the next chunk.
  2. You need to prevent an attacker truncating the entire file. If you can calculate the encrypted file length in advance, it can be converted to bytes and used as associated data for the first chunk. However, a more common solution is the STREAM construction, which involves a counter nonce with the last byte reserved to indicate whether it's the final chunk.

Here's some further reading:

$\endgroup$
5
  • $\begingroup$ This seems like a useful idea in WebCrypto API which doesn’t (yet) support a streaming cipher. For example it support AES-GCM but that needs the entire payload to authenticate. Breaking it up into chunks would allow AES to be used in modern browsers via WebStreams so it doesn’t need to load >1GB ArrayBuffers into memory! $\endgroup$
    – Sukima
    Commented Mar 18 at 22:19
  • $\begingroup$ If using AES-GCM for each chunk, can a hash of the previous authenticated chunk be used as the next chunk’s IV? $\endgroup$
    – Sukima
    Commented Mar 18 at 22:20
  • $\begingroup$ Is it possible to perform this with the current WebCrypto API? $\endgroup$
    – Sukima
    Commented Mar 18 at 23:54
  • $\begingroup$ @Sukima A counter is the ideal way. Hashing the previous chunk would be expensive, and the AES-GCM nonce isn't very big. You could do something like using the previous tag as associated data, rekeying for every chunk using a KDF, or libsodium's nonce XOR part of the tag alongside a counter. Streaming encryption should be possible to implement with the WebCrypto API. $\endgroup$ Commented Mar 19 at 18:11
  • $\begingroup$ I’m actually really surprised that this proposal does not offer any hints to a ponyfill/polyfill/implementation pseudo code considering all the parts are there just not plugged together. Has anyone out there done this yet? $\endgroup$
    – Sukima
    Commented Mar 19 at 21:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.