Any good file format alternative to PGP for encrypting data at rest?

Question

I'd like to create an encrypted file that:

• Can be decrypted on a variety of platforms (mainly Windows, OS X, and Linux) with knowledge of the key.
• Can be decrypted with existing tools (perhaps writing a shell script piping together common tools).
• Is in such a simple file format such that it'd be relatively easy to write your own decryption function from scratch assuming you had a standard library that included primitives like AES and HMAC-SHA256 without spending days or weeks trying to understand the file format.

So far, I've come up with three alternatives:

1. PGP - This sounds like a no-brainer, but its file format is quite complex and would require using a library like BouncyCastle. I'd rather have something simpler and not need dependencies.
2. OpenSSL - It has a very simple file format. I was able to write my own reader and writer in a few lines of code after understanding its key and IV derivation function. However, the format doesn't have a MAC in its standard.
3. KeyCzar - The encrypted file format does have a MAC in it, but the tools to encrypt and decrypt files aren't nearly as popular as PGP or OpenSSL.

Given these options, OpenSSL sounds very compelling if I could incorporate the MAC in it in an intuitive way. I could run openssl dgst on the encrypted file and then concatenate it to the end of it, but that is a bit hacky and I'm worried that it might be easy to mess up the Encrypt-then-MAC characteristics of the file in custom implementations.

Am I missing an easy answer to this problem?

Thanks in advance.

Answer 1

The more I think about this, the more I think it'd be better to not do this.

I couldn't think of a single file format that would be simple enough for this - they all have atleast some structure that is hard to replicate via shell-scripts and the like. Also, security considerations for the file format crop up very easily, especially if taking passwords in to consideration as well.

From the existing utilities, I suggest a quick look at scrypt. It has a pretty simple on-disk file format and meticulous attention to detail in general. It doesn't do encryption with actual keys, only passwords, though. The problem with that is that the scrypt key derivation function uses Salsa20 internally and isn't that simple to reimplement.

But, never the less, I thought I'd create something akin to what you wanted, but without any password functionality.

• genkey.sh:

#!/usr/bin/sh

openssl rand -out $1 64

• enc.sh:

#!/bin/sh

KEY=$1
FILE=$2
OUTPUT=$3

HMAC_KEY=`hexdump -v -e '/1 "%02x"' -n 32 -s 0 $KEY`
AES256_KEY=`hexdump -v -e '/1 "%02x"' -n 32 -s 32 $KEY`

openssl rand -out $OUTPUT 16
AES256_IV=`hexdump -v -e '/1 "%02x"' -n 16 $OUTPUT`
openssl enc -in $FILE -K "$AES256_KEY" -iv "$AES256_IV" -aes-256-cbc >> $OUTPUT
openssl dgst -hmac $HMAC_KEY -binary -sha256 $OUTPUT >> $OUTPUT

• dec.sh:

#!/bin/sh

KEY=$1
FILE=$2
OUTPUT=$3

HMAC_KEY=`hexdump -v -e '/1 "%02x"' -n 32 -s 0 $KEY`
AES256_KEY=`hexdump -v -e '/1 "%02x"' -n 32 -s 32 $KEY`
AES256_IV=`hexdump -v -e '/1 "%02x"' -n 16 -s 0 $FILE`

SIZE=`stat -c"%s" $FILE`
HMAC_BLOCKS=$(($SIZE/16 - 2))
AES_BLOCKS=$(($SIZE/16 - 3))

HMAC_FILE=`dd if=$FILE bs=16 skip=$HMAC_BLOCKS | hexdump -v -e '/1 "%02x"'`
HMAC_CALCULATED=`dd if=$FILE bs=16 count=$HMAC_BLOCKS | openssl dgst -hmac $HMAC_KEY -sha256 -binary | hexdump -v -e '/1 "%02x"'` 
if [ "x$HMAC_FILE" != "x$HMAC_CALCULATED" ]; then
  echo "fuuuuu"
else
  dd if=$FILE bs=16 skip=1 count=$AES_BLOCKS status=noxfer | openssl enc -d -K "$AES256_KEY" -iv "$AES256_IV" -aes-256-cbc -out $OUTPUT
fi

Key format is:

• 32 bytes AES-256 key
• 32 bytes HMAC-SHA256 key

Encrypted file format is:

• 16 bytes AES IV (random generated each time)
• x*16 bytes AES-256-CBC encrypted data with PKCS#5 padding
• 32 bytes of HMAC-SHA256 over the entire preceding file (including IV and padded and encrypted data)

The HMAC is verified first, so no data is decrypted in case the MAC is wrong. The IV is included in the HMAC, so there's no monkey business with that either. To get a less than 2^-32 probability of a duplicate IV, the same key can be used a maximum of 2⁴⁸, which should be plenty. A single file is probably limited in length by the HMAC to be maximum of 2⁶⁰ - 1 bytes, which should also be plenty. There should be no attacks allowing forgery, recovery of the key or recovery of the plaintext under 2¹²⁸ complexity. The entire resulting file should be indistinguishable from random except for the fact that the file size is always a multiple of 16. But no guarantees about security are given, peer review is a must.

The scripts do no argument validation and are in general written just to show how they work, not for safety or actual use. In particular, the actual encryption keys show up in command-line arguments and hence can be seen with ps by any user - so these really shouldn't be used for anything serious.