I'd like to create an encrypted file that:

  • Can be decrypted on a variety of platforms (mainly Windows, OS X, and Linux) with knowledge of the key.
  • Can be decrypted with existing tools (perhaps writing a shell script piping together common tools).
  • Is in such a simple file format such that it'd be relatively easy to write your own decryption function from scratch assuming you had a standard library that included primitives like AES and HMAC-SHA256 without spending days or weeks trying to understand the file format.

So far, I've come up with three alternatives:

  1. PGP - This sounds like a no-brainer, but its file format is quite complex and would require using a library like BouncyCastle. I'd rather have something simpler and not need dependencies.
  2. OpenSSL - It has a very simple file format. I was able to write my own reader and writer in a few lines of code after understanding its key and IV derivation function. However, the format doesn't have a MAC in its standard.
  3. KeyCzar - The encrypted file format does have a MAC in it, but the tools to encrypt and decrypt files aren't nearly as popular as PGP or OpenSSL.

Given these options, OpenSSL sounds very compelling if I could incorporate the MAC in it in an intuitive way. I could run openssl dgst on the encrypted file and then concatenate it to the end of it, but that is a bit hacky and I'm worried that it might be easy to mess up the Encrypt-then-MAC characteristics of the file in custom implementations.

Am I missing an easy answer to this problem?

Thanks in advance.

  • 3
    $\begingroup$ Do you want symmetric encryption only? Is the thing you are encrypting with going to be a password/passphrase or an actual key? $\endgroup$
    – Nakedible
    Aug 19, 2011 at 18:55
  • $\begingroup$ I only care about symmetric ciphers, and realistically only AES 128 and AES256. I was thinking the default file extension would be ".aes128" and ".aes256". My key will be random bytes the size of what the cipher expects. However, it'd be nice to also have a passphrase to key bytes function specified. Regardless, the file format needs the IV as I'll probably use CBC mode. $\endgroup$
    – Jeff Moser
    Aug 19, 2011 at 19:18
  • 1
    $\begingroup$ Your file format does not need an IV if the same key is only used to encrypt a single file. If the key is shared, you need to make sure the IV isn't repeated. Also, if you have a separate MAC for authentication, it needs a separate key - or you have to specify a key derivation function as well. You could get away with sharing the key, but you probably shouldn't do it. For passphrase to key bytes function, you need to decide how long it can take. 5 seconds of PBKDF2 is probably good, but there's probably no command-line utility for it... $\endgroup$
    – Nakedible
    Aug 19, 2011 at 19:38
  • $\begingroup$ I was just wanting to use whatever is common already so that I can write a simple script to decrypt it if things like OpenSSL are already available. Is it best to just use an HMAC on the existing OpenSSL file along with the key? I'd then just need a simple way with unix tools to strip that hash off the end and then verify it. Right now I might be leaning towards something like [OpenSSL Salted file encrypted with key X, HMAC256(key bytes || IV || encrypted file contents)] but am open to better ideas $\endgroup$
    – Jeff Moser
    Aug 19, 2011 at 20:00
  • $\begingroup$ @Nakedible, omitting the IV is pretty dangerous. I wouldn't recommend it. You just know that some user will use the same key twice inadvertently, and get screwed. Good point about the need for a key derivation function. (Sharing the key is dangerous.) $\endgroup$
    – D.W.
    Aug 20, 2011 at 1:55

1 Answer 1


The more I think about this, the more I think it'd be better to not do this.

I couldn't think of a single file format that would be simple enough for this - they all have atleast some structure that is hard to replicate via shell-scripts and the like. Also, security considerations for the file format crop up very easily, especially if taking passwords in to consideration as well.

From the existing utilities, I suggest a quick look at scrypt. It has a pretty simple on-disk file format and meticulous attention to detail in general. It doesn't do encryption with actual keys, only passwords, though. The problem with that is that the scrypt key derivation function uses Salsa20 internally and isn't that simple to reimplement.

But, never the less, I thought I'd create something akin to what you wanted, but without any password functionality.

  • genkey.sh:

openssl rand -out $1 64
  • enc.sh:


HMAC_KEY=`hexdump -v -e '/1 "%02x"' -n 32 -s 0 $KEY`
AES256_KEY=`hexdump -v -e '/1 "%02x"' -n 32 -s 32 $KEY`

openssl rand -out $OUTPUT 16
AES256_IV=`hexdump -v -e '/1 "%02x"' -n 16 $OUTPUT`
openssl enc -in $FILE -K "$AES256_KEY" -iv "$AES256_IV" -aes-256-cbc >> $OUTPUT
openssl dgst -hmac $HMAC_KEY -binary -sha256 $OUTPUT >> $OUTPUT
  • dec.sh:


HMAC_KEY=`hexdump -v -e '/1 "%02x"' -n 32 -s 0 $KEY`
AES256_KEY=`hexdump -v -e '/1 "%02x"' -n 32 -s 32 $KEY`
AES256_IV=`hexdump -v -e '/1 "%02x"' -n 16 -s 0 $FILE`

SIZE=`stat -c"%s" $FILE`
HMAC_BLOCKS=$(($SIZE/16 - 2))
AES_BLOCKS=$(($SIZE/16 - 3))

HMAC_FILE=`dd if=$FILE bs=16 skip=$HMAC_BLOCKS | hexdump -v -e '/1 "%02x"'`
HMAC_CALCULATED=`dd if=$FILE bs=16 count=$HMAC_BLOCKS | openssl dgst -hmac $HMAC_KEY -sha256 -binary | hexdump -v -e '/1 "%02x"'` 
if [ "x$HMAC_FILE" != "x$HMAC_CALCULATED" ]; then
  echo "fuuuuu"
  dd if=$FILE bs=16 skip=1 count=$AES_BLOCKS status=noxfer | openssl enc -d -K "$AES256_KEY" -iv "$AES256_IV" -aes-256-cbc -out $OUTPUT

Key format is:

  • 32 bytes AES-256 key
  • 32 bytes HMAC-SHA256 key

Encrypted file format is:

  • 16 bytes AES IV (random generated each time)
  • x*16 bytes AES-256-CBC encrypted data with PKCS#5 padding
  • 32 bytes of HMAC-SHA256 over the entire preceding file (including IV and padded and encrypted data)

The HMAC is verified first, so no data is decrypted in case the MAC is wrong. The IV is included in the HMAC, so there's no monkey business with that either. To get a less than 2-32 probability of a duplicate IV, the same key can be used a maximum of 248, which should be plenty. A single file is probably limited in length by the HMAC to be maximum of 260 - 1 bytes, which should also be plenty. There should be no attacks allowing forgery, recovery of the key or recovery of the plaintext under 2128 complexity. The entire resulting file should be indistinguishable from random except for the fact that the file size is always a multiple of 16. But no guarantees about security are given, peer review is a must.

The scripts do no argument validation and are in general written just to show how they work, not for safety or actual use. In particular, the actual encryption keys show up in command-line arguments and hence can be seen with ps by any user - so these really shouldn't be used for anything serious.

  • $\begingroup$ Thanks for your thoughtful write-up! A few thoughts: 1. I was assuming I'd use OpenSSL such that you'd have to enter the key via stdin instead of the command line. 2. Since the keys would effectively be entered in manually, I wanted to keep the key size as small as possible. The only way I see to reconcile it is to have a key derivation function that expanded the key into the actual cipher key and then the HMAC key. 3. I hadn't considered scrypt. Would you recommend it over PGP for encrypting single files in terms balancing simplicity but still having security? (ignoring Salsa20 issues?) $\endgroup$
    – Jeff Moser
    Aug 19, 2011 at 22:33
  • $\begingroup$ Nice design, Nakedible! Looks well thought-out to me. And I agree with your overall recommendation: probably better not to do something like this, all told. $\endgroup$
    – D.W.
    Aug 20, 2011 at 1:57
  • $\begingroup$ @Jeff, yes, you could use a key derivation function to expand the user-typed-in key to a cipher key and HMAC key. e.g., cipher-key = SHA256(K || "cipher-key"), hmac-key = SHA256(K || "hmac-key"). There are usability issues with user-typed-in keys, though. If users are typing in keys, I recommend generating K from the user-typed-in value via bcrypt, scrypt, or PBKDF2, with a suitably large number of iterations; I suspect users will be tempted to type in a passphrase of their own choosing, which is a significant danger (especially if you don't use bcrypt or similar). $\endgroup$
    – D.W.
    Aug 20, 2011 at 2:01
  • $\begingroup$ If you have manual key entry, then you have to be prepared for typing errors. This means that typing errors should usually be caught before going through the entire file. This means that there should be some pre-verifier for the key only. This, combined with the need for some key derivation, complicates the protocol enough that I would definitely recommend sticking with a "standard" file format, even though they are complex. $\endgroup$
    – Nakedible
    Aug 20, 2011 at 12:04
  • $\begingroup$ I would consider the requirement to process the entire file as being a bonus instead of disadvantage. That would be a serious deterregent for a would-be-attacker. $\endgroup$ Apr 13, 2020 at 17:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.