Future-Proof Versioning and Validation

Question

I am working on a library (using standard primitives: AES256 CTR; HMAC with SHA256; PBKDF2 with SHA256, 128 bit salt, and 10000 rounds) to encrypt and decrypt data, given a password.

The encrypted data includes a header with version information and an HMAC (the header is included in the HMAC).

I am worried about exactly how I should check/use HMAC and version:

In a future release (with a new version number), the HMAC algorithm or parammeters may have changed. So I need to check version first and abort if the version is newer than the current code. But that means that if someone maliciously alters the version then they can "trick" the code into aborting (and likely issuing a warning to the user saying that they should update their software).

Alternatively, in the future, if the version is within acceptable range, then I need to use that to select the HMAC. That will then validate the version. But it seems like it could lead to an attack that allows an older, broken HMAC to be abused in some way.

Is there some standard way around these issues? Or are they not considered important? Or do I simply need to wait until I have more specifics (for the second case). Is there anything else I should know? Should I extend the API now to include a flag that controls backwards compatibility? Since the API is meant to be very simple that needs a default - what should it be?

I realise that I am not the best person in the world to be doing this work, but no similar library appears to exist for Python 3 and so far I have been lucky enough to find useful support from others (eg on HN; also note that I am not writing any algorithmic code - everything is delegated to pycrypto). Thanks.

Answer 1

You could have "protocol ID's" which describe the set of primitives and parameters each library version is capable of using. You may deprecate older ID's once they become obsolete by just removing them from some future version, this way you can reject files which are deemed too old (or too recent) and thus unsupported, as long as you don't change how you read the ID's.

To illustrate, suppose you have your initial v1 library, with this sole valid hash/cipher/kdf combination:

<MD4, DES-CBC, MyHomemadeKDF> = 0x01

Thus, because this is the only combination available, all files encrypted with v1 will bear the protocol ID 0x01 (somewhere before the header, in plaintext, next to the file version) and this is recognized by v1.

You then realize this is a pretty poor choice of algorithms, and decide to deprecate it. You create your v2 library, remove 0x01 from the protocol list, and add 0x02 and 0x03:

<SHA256, AES-128-CTR, PBKDF2/128/10000> = 0x02
<SHA512, AES-256-CTR, PBKDF2/128/50000> = 0x03

At this point, any attempt to feed it a file encrypted with your v1 will fail, because its ID is 0x01, which is not recognized by v2 as a valid protocol. Similarly, trying to give a v2-encrypted file to your v1 library will also fail, since the file is using a more recent protocol which v1 does not recognize.

Now, an attacker can obviously change the version number, and cause the software to fail. This cannot be avoided, and in this respect is not important, but can be detected easily: if the attacker changes the protocol ID to an invalid one, the software will complain the file is incompatible, and on the other hand, if he does change the protocol ID to a valid one (say 0x02 to 0x03, for v2) then decryption will clearly fail since you're using the wrong algorithms. You can implement a sanity check in the form of an extra HMAC if you really want to make sure, something like an extra header field:

$$\mathrm{HMAC}(K, \mathrm{ProtocolID} ~ || ~ \mathrm{FileVersion})$$

Which the library can use to verify the ID or file version were not tampered with ($K$ is the encryption, or a derived, key). Either the attacker changes those, or he tries to mess with the encrypted header and HMAC's, he gets busted either way. This way he should not be able to manipulate anything.

This is similar, in essence, to the "cipher suite" concept used by SSL/TLS, which uses them to negociate which algorithms should be used for key exchange and secure communication. You could use a plugin system to transparently let the user decide which set of primitives he would like to accept or reject.

The ID's don't have to be monotonically increasing, as long as you don't reuse them, and so you might prefer deterministically deriving them from the names of the primitives involved just for robustness.

If you end up doing something like this, make sure to provide facilities to upgrade files from one version to the next so you don't confuse users who still have files encrypted with old algorithms.

---

However, as you want simplicity, this may be overkill as a full-blown version management scheme. How simple does your API need to be? Since your library seems to be very high-level, I imagine you'll have a small toolchain of encryption and decryption programs to go with it eventually? Like OpenSSL's enc?