I am working on a library (using standard primitives: AES256 CTR; HMAC with SHA256; PBKDF2 with SHA256, 128 bit salt, and 10000 rounds) to encrypt and decrypt data, given a password.

The encrypted data includes a header with version information and an HMAC (the header is included in the HMAC).

I am worried about exactly how I should check/use HMAC and version:

In a future release (with a new version number), the HMAC algorithm or parammeters may have changed. So I need to check version first and abort if the version is newer than the current code. But that means that if someone maliciously alters the version then they can "trick" the code into aborting (and likely issuing a warning to the user saying that they should update their software).

Alternatively, in the future, if the version is within acceptable range, then I need to use that to select the HMAC. That will then validate the version. But it seems like it could lead to an attack that allows an older, broken HMAC to be abused in some way.

Is there some standard way around these issues? Or are they not considered important? Or do I simply need to wait until I have more specifics (for the second case). Is there anything else I should know? Should I extend the API now to include a flag that controls backwards compatibility? Since the API is meant to be very simple that needs a default - what should it be?

I realise that I am not the best person in the world to be doing this work, but no similar library appears to exist for Python 3 and so far I have been lucky enough to find useful support from others (eg on HN; also note that I am not writing any algorithmic code - everything is delegated to pycrypto). Thanks.

  • 2
    $\begingroup$ What means "the HMAC may have changed"? Do you mean the selection of the algorithms and parameters might have changed? $\endgroup$ Dec 27, 2012 at 22:00
  • $\begingroup$ "But that means that if someone maliciously alters the version then they can "trick" the code into aborting." If they are able to modify "packets", then they already can trick your code into aborting in any case by sending garbage or holding packets to make receiver believe that sender didn't send anything. $\endgroup$
    – dchest
    Dec 27, 2012 at 23:44
  • $\begingroup$ @PaŭloEbermann - yes, clarified $\endgroup$ Dec 27, 2012 at 23:47
  • $\begingroup$ @dchest - true, but in general corrupting the data will simply abort things. changing the version number triggers a more specific action (update your software) that might be combined with some kind of attack where a fake replacement is provided. i realise i may be taking things too far, but it seemed like it was different... $\endgroup$ Dec 27, 2012 at 23:49
  • 1
    $\begingroup$ @andrew cooke: Respectfully, perhaps you should consider contributing to Google Keyczar instead. Programmers need a small number of extremely well-tested crypto libs, because crypto is fiendishly difficult to implement. Here is an example from Keyczar: rdist.root.org/2009/05/28/… $\endgroup$
    – user950
    Dec 30, 2012 at 11:38

1 Answer 1


You could have "protocol ID's" which describe the set of primitives and parameters each library version is capable of using. You may deprecate older ID's once they become obsolete by just removing them from some future version, this way you can reject files which are deemed too old (or too recent) and thus unsupported, as long as you don't change how you read the ID's.

To illustrate, suppose you have your initial v1 library, with this sole valid hash/cipher/kdf combination:

<MD4, DES-CBC, MyHomemadeKDF> = 0x01

Thus, because this is the only combination available, all files encrypted with v1 will bear the protocol ID 0x01 (somewhere before the header, in plaintext, next to the file version) and this is recognized by v1.

You then realize this is a pretty poor choice of algorithms, and decide to deprecate it. You create your v2 library, remove 0x01 from the protocol list, and add 0x02 and 0x03:

<SHA256, AES-128-CTR, PBKDF2/128/10000> = 0x02
<SHA512, AES-256-CTR, PBKDF2/128/50000> = 0x03

At this point, any attempt to feed it a file encrypted with your v1 will fail, because its ID is 0x01, which is not recognized by v2 as a valid protocol. Similarly, trying to give a v2-encrypted file to your v1 library will also fail, since the file is using a more recent protocol which v1 does not recognize.

Now, an attacker can obviously change the version number, and cause the software to fail. This cannot be avoided, and in this respect is not important, but can be detected easily: if the attacker changes the protocol ID to an invalid one, the software will complain the file is incompatible, and on the other hand, if he does change the protocol ID to a valid one (say 0x02 to 0x03, for v2) then decryption will clearly fail since you're using the wrong algorithms. You can implement a sanity check in the form of an extra HMAC if you really want to make sure, something like an extra header field:

$$\mathrm{HMAC}(K, \mathrm{ProtocolID} ~ || ~ \mathrm{FileVersion})$$

Which the library can use to verify the ID or file version were not tampered with ($K$ is the encryption, or a derived, key). Either the attacker changes those, or he tries to mess with the encrypted header and HMAC's, he gets busted either way. This way he should not be able to manipulate anything.

This is similar, in essence, to the "cipher suite" concept used by SSL/TLS, which uses them to negociate which algorithms should be used for key exchange and secure communication. You could use a plugin system to transparently let the user decide which set of primitives he would like to accept or reject.

The ID's don't have to be monotonically increasing, as long as you don't reuse them, and so you might prefer deterministically deriving them from the names of the primitives involved just for robustness.

If you end up doing something like this, make sure to provide facilities to upgrade files from one version to the next so you don't confuse users who still have files encrypted with old algorithms.

However, as you want simplicity, this may be overkill as a full-blown version management scheme. How simple does your API need to be? Since your library seems to be very high-level, I imagine you'll have a small toolchain of encryption and decryption programs to go with it eventually? Like OpenSSL's enc?

  • $\begingroup$ thanks. so apologies for not being clear enough that what seems to be the root of the problem is that i want to both (1) automatically support decrypting data encrypted with an older version and (2) somehow avoid attacks that supporting an older version could imply. i guess looked at that way i may be asking the impossible. to answer the final q - the api is ideally two functions, encrypt and decrypt. but maybe i need to consider conversion instead of automatic backwards support. ps the current heder includes what you are calling a protocol id. $\endgroup$ Dec 27, 2012 at 23:51
  • 1
    $\begingroup$ @andrewcooke Well, if the old algorithms are broken beyond repair, not much you can do about it, beyond asking your users to migrate their files to newer versions. Algorithms don't live forever. What attacks are you thinking about? $\endgroup$
    – Thomas
    Dec 27, 2012 at 23:54
  • $\begingroup$ i don't have a specific attack in mind (hence my comment that maybe i need to wait for more specifics). i'm just assuming i may have screwed up. thanks for the comments - it's given me more to think about. i'll wait a while to see if anyone else posts, but that was help enough for a "correct" answer, thanks. $\endgroup$ Dec 27, 2012 at 23:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.