I would like to encrypt and decrypt files with AES-256-GCM. I need a solution that works with Java, PHP and JavaScript. In practice, users can upload, download files via web browser and/or desktop app and there are files that will be generated by the server itself. (I'm currently using openSSL, but I'm thinking of switching to Sodium). I have a working solution already that works fine using relatively small files (<100MB). However, I am concerned about memory usage for large files (like videos, disk images).

I found a couple of implementations where file chunks are used. (The resources I found were not specifically security-related sites.)

The concept is:

  1. Read 1MB from the original file
  2. Generate an IV
  3. Encrypt the chunk
  4. Save the IV, ciphertext, and tag to the output file
  5. Repeat until the end of the file

The IV is always 12 byte and the tag is always 16 byte so it is not very difficult to implement it.

The solution works, but the real question is, is this solution good from a security standpoint or should something else be used?

  • $\begingroup$ This is a software problem that GCM implementations have. I had proposed chaining in SO. Keep in mind that one can continue the counter where it is left instead of using a new IV per chunk, though that may require to touch the source code of the libraries. Also, if you use SSL/TLS there is a limit a record $2^{14}$ bytes. $\endgroup$ – kelalaka Jun 25 '20 at 14:47
  • $\begingroup$ Do you plan to send the files over the network? $\endgroup$ – kelalaka Jun 25 '20 at 14:50
  • $\begingroup$ Right now, I’m primarily interested in security risks, so that’s why I posted here. And yes this is a webapp where users can upload and download files but there are files that are generated on the server. $\endgroup$ – Vmxes Jun 25 '20 at 14:54

No, the way that it is described, the solution is insecure.

The chunks themselves are of course secure. That is: as long as you verify the authentication tag and if the IV is indeed unique, i.e. a nonce as required by GCM.

However, an adversary can simply switch around the chunks in the file, including the IV and authentication tag. That way the file can be rearranged. Similarly, whole chunks can be removed without issue. If you use the same key for multiple files, you can replace one file with another, or even move chunks between files.

The way around this is to verify that all the authentication tags are in the correct order, e.g. by using HMAC or even by using them as Additional Authenticated Data (AD or AAD) in a separate call to GCM. Then add the final authentication tag to the start or end of the ciphertext of the chunks.

As GCM is insecure for large amounts of data, I'd use a separate data key for each file. You can generate one using a key based key derivation function (KBKDF such as HKDF) from the master key and a 256 bit random salt (the salt needs to be included with the ciphertext, of course). Then you can use a zero based 12 byte statically sized counter as nonce.

That will still leave the issue of being able to switch files. That could be avoided by including the file path into the calculation of the keys and saving it in the file, but beware that this will break any file that is renamed. This is a pretty tricky issue that will require domain specific solution. Similarly, changes to files are as tricky to handle. Some kind of separate register / third party may be required.

Note that in principle GCM uses CTR mode and can be made an online capable cipher. Such a cipher directly converts plaintext to ciphertext and vice versa. That way you can have, say, 1 GiB chunks while using a buffer size of 1 MiB or less (or even no application specific buffer at all).

Problem is that many implementations of GCM do away with this property. Some do allow incremental "update" calls, but even then the decryption specifically may not be fully online. An example of this is Java where you have calls to update methods, but since the authentication tag is considered part of the ciphertext the online property during decryption is lost (i.e. it has to buffer as many bytes as the authentication tag, because it cannot distinguish between the two until all the bytes are updated).

  • $\begingroup$ Thanks for the detailed explanation. What are the large amounts of data? Just to clarify the key generation. I have a 256bit master key that is constant and I generate a 256bit random salt for each file. And from these two 256bit entries will be converted (with HKDF) to one file specific 256bit key. Did I understand correctly? $\endgroup$ – Vmxes Jun 29 '20 at 9:21
  • $\begingroup$ Yeah, that's basically it. That way the amount of data to be encrypted with GCM can be controlled to a large extend, which is important for GCM. Generally files are limited in size (at least to some degree), but the amount of files is unknown. Still, you might want to look elsewhere if you include archives of blueray titles or modern games ;) $\endgroup$ – Maarten Bodewes Jun 29 '20 at 11:39
  • $\begingroup$ Even if Java 'standard' crypto (SunJCE) saw the tag separately from the ciphertext, it still can't complete authentication until processing all of the ciphertext, and the AE-with-or-without-AD 'contract' is not to release potential plaintext before authentication is complete, as vividly demonstrated by the 'efail' attack. The BouncyCastle Java provider does so anyway, needing to buffer only a few bytes in the same way as PKCS5/7-unpadding has long done. $\endgroup$ – dave_thompson_085 Oct 27 '20 at 10:31
  • $\begingroup$ I'm in principle against performing any operations on unauthenticated plaintext. However, I do think that there are reasons why you want to have a generic API. The tag could be stored elsewhere, and it makes sense to store decrypted message data on disk when decrypting files instead of storing both the ciphertext and plaintext in memory, then finally copy the plaintext file to the right location after validation. The fact that you should not process the data should be handled at the application layer IMHO. $\endgroup$ – Maarten Bodewes Oct 27 '20 at 11:02
  • $\begingroup$ Don't forget to accept answers if they resolve your issue. Or indicate why they don't, of course. $\endgroup$ – Maarten Bodewes Oct 27 '20 at 17:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.