# Is there a standard, or recommended format for encrypted data, with algorithmic agility?

I am storing relatively small (from a few bytes to less than 50MB) amounts of encrypted data for an extended period of time. This data is encrypted using a custom tool, and has been in use for some amount of time already. I am reviewing the current implementation, and am considering whether I need to make some changes. I must retain support for previously encrypted files and so (I think) must retain the leading Format ID in order to distinguish between formats. Beyond that, I am free to modify the format however I like.

If there is a standard or de facto format which would provide algorithmic agility I would greatly prefer that to rolling my own. Is there such a thing?

Alternatively, what standards/best practices apply to developing such a format? Are there particular pitfalls I should be mindful of?

The existing format currently is as follows:

## File format

# +--------------------------------+
# |           Format ID            |  1 byte
# +--------------------------------+
# |        Unused (Legacy)         |  3 bytes
# +--------------------------------+
# |           Cipher ID            |  2 bytes
# +--------------------------------+
# |            HMAC ID             |  2 bytes
# +--------------------------------+
# |      Initialization Vector     |  16 bytes
# |                                |
# +--------------------------------+
# |           Key Salt             |  8 bytes
# +--------------------------------+
# |           HMAC Tag             |  Variable length, varies with
# ...                            ...  digest algorithm
# |                                |
# +---------------------------------
# |        Encrypted Data          |
# |                                |
# |                                |  Variable length
# ...                            ...
# |                                |
# +---------------------------------


Should the HMAC Tag cover just the encrypted data and the IV (that is what it does currently). Is it desireable/practical to provide some integrity for the other parts of the header as well?

• You might want to checksum the header, but I can't think of any particular attack that could be made possible by modifying the header that an HMAC would protect from. – forest Jul 25 '19 at 2:04
• Also, note that this doesn't provide complete algorithmic agility. For example, you're stuck using block ciphers with a fixed block size and a specific mode of operation, and you're limited to using HMAC. What if you want to use ChaCha20-Poly1305? What if you wanted to use a block cipher with a 256-bit block size (so a 32 byte IV)? – forest Jul 25 '19 at 9:05
• @forest yeah, the format described is the current one. I'm considering what might be better. I'm considering combining cipher/hmac into a single field which represents a predefined set of cipher/hmac/whatever. The rest of the header would then vary based on the indicated cipher suite. – vezult Jul 25 '19 at 17:09

The layout of the format doesn't matter much. I would recommend combining all the parameters into a single cipher suite name so that you're not tempted to make nonsensical combinations: just use NaCl crypto_secretbox_xsalsa20poly1305 (or AES-GCM if you must use AES, with the caveat that it subjects your users to side channel attacks on software AES implementations), not your own composition of AES256-CTR with HMAC-SHA256.

What matters much more is:

• How many cryptographic decisions are imposed upon users who are not competent to answer questions on this site? Correct answer: none.

For example, although the scrypt file format has some options, there are no cipher fields—just a format version number. There are cost parameters for the password-based key derivation function, but when creating files, even those are automatically calibrated by the scrypt utility by default, with the option to specify them manually.

• How are the parties in the system to behave when fed instances of the format? Be careful to specify protocol, not just data. How do you use the key and act on the file?

If you are using a key for a particular purpose, say an RSASSA-PSS public verification key, do not let the format dictate another purpose. The implementation ecosystem of JavaScript Web Tokens, JWT, is a disaster because it was designed as a lofty view of data, abstracted from the reality of protocols.

JWT implementations often expose a generic verify(token, key) function which uses key however the attacker-controlled token prescribes. Even if you meant for key to be an RSASSA-PSS public verification key, token might say ‘hey, JavaScript chump, please use HMAC-SHA256!’, and you, the chump who used JWT, will happily use the ‘RSASSA-PSS public verification key’ in the place of an HMAC-SHA256 key which enables trivial forgery since RSASSA-PSS public verification keys are, well, often public.