How to handle authenticated encryption for big files?

Question

First I need to mention that I can't use BouncyCastle to implement this solution.

I have to encrypt files into the file system. Those files are not going to be tranfered over the wire to any other location. They are only going to be stored in the server for later use by an application installed there. The file system may or may not be encrypted at an OS level, but that's something that I can't count on.

My files are going to be big files (say from several hundreds of megs to some times a couple of GBs). And they will be created on a regular basis, so I can expect to have a growing number of them. Also, sometimes some of this files are going to be deleted cause they may no longer be needed, and some other times files will be kept for years.

I already have the encryption in place, with AES/CBC/PKCS5Padding, using a 256 bit key and I/O Cipher Streams. I have no authentication thought.

I don't think I can assume that it is safe to just encrypt and not use authentication as the server could be hacked I guess and the files stolen.

Also, by reading several questions it's clear that no authentication is just someting I shouldn't even think of.

So what are my options here then? I see 2 with some pro/cons analysis:

Option 1: AES/CBC + MAC (say with SHA-256)

Pros

• Already have the encryption part working
• Supported by Java with no extra libs

Cons

• I have to deal with the computation of the MAC
• I have to deal with appending the mac to the file and extracting it later
• I will have to handle validation myself.
• Lots of room for me not getting that right I'm affraid.

Option 2: Use GCM operation mode

Pros

• Forget about handling tag computation / message authentication
• Supported by Java (I'm using 1.8, so the default provider supports it)

Cons

• I found that there are people who wouldn't recommend it.
• Also this post seems to suggest that performance is somewhat an issue with Java8, but that's something I haven't verified (Update: @archie pointed to here and here that comfirm the performance issues).
• It is badly broken in Java 1.7. I have yet to verify it works ok in Java 1.8 (update: this thread verifies that problems are still present in Java 8)
• I'm using a single symmetric key for encryption. I'm worried about how to manage the nonce so I don't break the security properties of the algorithm (My current implementation gets the IV from cipher.getIV() after the Cipher is initialized. I'm not sure if that's good enough for GCM or if I need to take other special considerations.)

Are those options that I mention valid? What about the pro/con analysis? Is there any other suggestion you would have? Which approach would you choose?

Answer 1

If you definitely can't use other libraries such as Bouncy Castle, your best option is probably to just add an HMAC authentication tag (it should be easy to write a MacInputStream/MacOutputStream to chain it into your existing pipeline).

The primary reason is that the JDK 8 (at least up to 1.8.0_25) GCM implementation suffers from two problems that make it impractical for large files:

• The implementation buffers the entire plaintext on decrypt (this is intended to prevent release of unauthenticated plaintext, but cripples the cipher implementation for uses that handle that at a higher level in the security protocol)
• The performance is crippled (3MB/s on my server) due to a naive multiplier implementation and no hardware acceleration for the GCM calculations (AES in JDK 8 is AES-NI accelerated and very quick, but AES/GCM is completely dominated by the broken GCM performance). There are intentions to implement hardware acceleration, and there have been third party submissions to improve the performance with, but these haven't made it to a release yet.

If you go with the AES/CBC + HMAC option, you'll also want to make sure you have independent keys for the encryption and Mac (e.g. using something like HKDF to generate the two keys from a single master key).

If you can use Bouncy Castle (even using the lightweight API if your restriction is that you can't use another JCE provider) then the OCB, GCM and EAX implementations in that will give you better performance (and take away a bunch of the manual fiddling you'll have to do with the HMAC approach).

Answer 2

Actually, you might be better of without any authentication.

But why? Everyone recommends it. Truth isn't so simple.

When we define our problem we define attackers we want to stop.

• You cannot stop your attacker from learning data you store when he is on your machine (when you can decrypt it, he can too).
• You cannot stop attacker from manipulating data on fly, or just learning HMAC key and using it to authenticate data for you.

So you, if you want to encrypt files on the server then authentication might be pointless. Encryption at all might be pointless if you store key on that server anyway.

So please define who you are trying to protect that data from. It might be pointless to do HMAC or any encryption in your case. Also, as a side note, when your files are big, you might consider AES-CTR since it has parallelization on both encryption and decryption.