In general, how do e2e messaging platforms deal with changes to the structure of the encrypted data?

Question

I'm a layman trying to deepen my understanding of crypto and private messaging by building a (proof-of-concept) centralized, end-to-end encrypted messaging "platform".

Messages are sent by devices to the "server" (running on my Pi inside my home network) where they are stored and from whence they can be retrieved by the recipients via an api call. The server knows just the bare minimum to distribute the messages correctly while the message text is encrypted and the whole message is signed properly.

I am now faced with the following challenge: I want to rename a property of the message format used by the client applications (think JSON: message.body → message.content). The problem now is, that (1) all the previous messages are stored on the server in the old format and (2) an "old" client may try to message a "new" client which forces me to somehow resolve this mismatch between the communicated messages' formats.

If the private keys were available to the server, this would be a trivial problem: simply transform the message to the recipient's expected message format. But since I'm trying to be truly private and the private keys are only know to their respective owners this isn't an option. The messages cannot be modified by the server.

I feel that at the heart of this problem there lies a more fundamental challenge which I'm sure people much more capable that I have come up with solutions and for which there exist standardized approaches. Thus here's my question: In general, how do e2ee messaging solutions (e.g. signal, matrix) handle changes to the structure of encrypted data? What approaches do exist except cutting off old versions and throwing away any message history?

I beg your pardon if this question sounds vague or is too broad. I'm just a noob trying to get the big picture.

Thanks a lot.

Answer 1

This is the generic IT problem of backward compatibility, made harder by crypto.

In IT, a common way to deal with backward compatibility is having a version field at start of data structures, allowing applications to decide if they recognize the format, and handle it if they can. New applications handle old formats, at least during a transition period. This can be extended to cryptographic payloads by having the version field in clear, and the rest encrypted/authenticated according to the version field.

When the method to encrypt/authenticate does not change, there is no crypto-specific issue, and usual mechanisms to deal with enriching data structures without compatibility break will do (e.g. in JSON, it's possible to add new fields that old applciations will skip). In the following I assume the crypto has changed, and a version field allows to detect that.

The problem now is, that (1) all the previous messages are stored on the server in the old format and (2) an "old" client may try to message a "new" client which forces me to somehow resolve this mismatch between the communicated messages' formats.

If the new client can still decipher the old format, neither is an issue. That's the way to go!

The real hard problem is when a "new" client sends a message. There are many choices, neither of which is perfect. Some of these:

• New clients always use the new format, which old clients won't understand.
• New clients use the old format until some trigger (switch date, message from server), then use the new format. Old clients won't be usable after the switch, and new clients won't enjoy the new features of the new format until that event.
• New clients use new format after the trigger or if there is some clue the receiver (or all the receivers) understands it, e.g. because the sending client has received an authenticated message from (all) the recipient(s) that indicates support of the new format, perhaps implicitly by using it.
• New clients generate both formats until the trigger and send both to the server, which stores them; old clients fetch the old format, new clients fetch the new and in case of failure fetch the old format. The server must handle requests in both formats, and needs more storage and more traffic in the interim period.