I have to encrypt big files. Say their size ranges from 500mb to several of gigabytes.

I would like to use AES/GCM/NoPadding as provided by Java 1.8 since that gives me automatic authentication and encryption.

I would like to use the handy Cipher Input/Output Stream cause I can chain it with GZip I/O Streams to compress the data a little before encryption.

However I was reading that the implementation of Java appends the authentication tag at the end of the stream. That means that for a long file, if I were using CipherInputStream to decrypt it, it wont be able to tell whether the contents have been tampered or not until it reaches the end of the stream, right?

If that were the case, wouldn't it be problematic to actually use that operation mode for what I'm trying to accomplish becuase since the file can't be decrypted in memory it will have to be decrypted somewhere in the filesystem until the failure is detected, leaving some time for an attacker to see the plain text?

Is this a potential thread and a real concern that I should be worrying about or there's something about the algorithm / cipher streams that I'm missing and prevents that from ocurring?

  • 1
    $\begingroup$ In general it is better to ask runtime specific questions at stackoverflow. I've not voted to close as your actual question is not directly about the library. I've split my answer up in a theoretical and a Java specific part though (maybe other answers, if any, could do the same). $\endgroup$ – Maarten Bodewes Nov 19 '14 at 12:53
  • $\begingroup$ I'd stay away from the JDK CipherInputStream as it will silently eat any AEAD validation failures. See JDK-8012900 and JDK-8016171. The JDK team know about this, but haven't been persuaded to change that behaviour (see this thread). Bouncy Castle has a more sane replacement implementation of CipherInputStream. $\endgroup$ – archie Nov 19 '14 at 21:24

There is nothing in the GCM cipher that prevents it's use it in streaming mode. You should however not use the resulting plaintext during decryption for anything that requires security before you have verified the authentication tag.

The authentication tag is not to prevent you from decrypting the ciphertext. It is there to provide for integrity and authenticity. You should never decrypt where an attacker can see the plaintext. If possible, you should even try and make it hard for an attacker to perform side channel analysis.

Note that GCM is bounded to encrypting about 68 GB ($2^{39} - 256$ in bits) of data for a single IV. The amount of invocations is $2^{32}$ but you should be advised to stay well away from those limits. Note that repeating the IV for two separate encryption invocations is a catastrophic event for GCM.

CipherInputStream in general is horrible. I would suggest to reprogram it using Cipher and memory mapped files and ByteBuffer itself. The Java implementation (where the tag is automatically put at the end) and CipherInputStream make for this horrible buffering mess.

I'm rewriting the Bouncy Castle implementation and I see a code & complexity reduction of about 30% when I separate the tag from the decryption, plus it enables to decrypt each byte separately. In other words it restores the online properties of the underlying CTR cipher.

With Java 8 however you may want to stick to the Java 8 implementation as GCM may be sped up using intrinsics (for the server VM on the latest Intel processors). Note that according to archie below this functionality is not yet present.

  • $\begingroup$ Beware that the tag size matters with GCM. You could think of EAX or HMAC based authentication if you don't want to look into the security of GCM. $\endgroup$ – Maarten Bodewes Nov 19 '14 at 10:36
  • 3
    $\begingroup$ Bear in mind that the JDK 8 GCM implementation has crippled performance due to a naive (no HW or even tables) multiplier implementation. It also buffers plaintext during decryption in memory, so is unusable for large files. Hardware acceleration and/or other multiplier improvements are apparently in the works, but not there as of 1.8.0_25. $\endgroup$ – archie Nov 19 '14 at 21:16
  • $\begingroup$ @archie Thanks for the warning. Note that you can always place the BC provider on top of the list of providers and have that take over, or make the provider configurable. I'll take another look at GCM as in Java 8, also because of a few updates regarding bugs with CipherInputStream. This is brand new functionality and is still a bit in movement. $\endgroup$ – Maarten Bodewes Nov 19 '14 at 23:20
  • $\begingroup$ @owlstead When you say that GCM is bound to ~68 GB, and that keys should be changed, my understanding is that it means to 68 GB in total, not necessarily on the same file. Is that the correct understanding? $\endgroup$ – alejo Nov 21 '14 at 0:13
  • 1
    $\begingroup$ @alejo: ~68 GB per IV ... so basically it's a per file limitation since one should always use a new truly random IV per file $\endgroup$ – DeepSpace101 Jun 2 '16 at 18:22

This is a common problem which most software doesn't handle, surprisingly. Yes, decrypting to the filesystem is problematic as you guessed.

A better approach would be to split the file in "packets" and encrypt/authenticate each packet separately. However you will have to take care of many details (e.g. you can write only the first IV and implicitly increment it for the next packets; you will need to write the tag for each packet; and you will need to somehow mark the last packet using an e.g. additional data field in order to prevent an attacker from extending or truncating the ciphertext; there may be other steps I'm not aware of). Sadly I don't know any common/standard way to do this in order to guide implementors.

  • $\begingroup$ Thanks. I've been trying to figure out this for a while. All I had the impression (due to my lack of experience with that) that there were going to be issues (the ones I mention on my question). Unfortunately I'm confirming that there are many caveats like those you mention. That makes me wonder whether the approach I'm trying to use is the appropriate or what's the way you would tackle this issue instead $\endgroup$ – alejo Nov 24 '14 at 16:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.