# Basic encrypted custom protocol

I've been suggested to ask here instead of stackoverflow, so here I am.

I'd like to encrypt the data of my application that uses a custom protocol I've created. I know about TLS and other protocols but I just want to do a basic encryption. The following code is C++, please ask me if you need any help to understand it.

My protocol is something like:

template<uint16_t thePayloadLength>
uint16_t startOfFrame = 0xFFFF; // Protocol signature
uint16_t version;
uint8_t src;
uint8_t dst;
}; // the template part probably is going to disappear (too much code bloat)


My idea is to create two functions like:

template<uint16_t thePayloadLength



that basically just do an encryption / decryption of my protocol struct. Then I can send the "EncryptedData" as a payload of an header like the following:

struct EncryptedProtocol{
uint16_t startOfFrame = 0xAAAA;
uint16_t version;
uint16_t sequenceNumber; // can be useful for UDP
uint16_t length; // lenght of the encrypted payload
};


I was thinking to use AES-GCM as encryption algorithm with the shared key already known by all the applications.

What do you think about it?

[EDIT] all the applications are outside Internet and they all know the public keys of all the other applications.

[EDIT2]

would you only feed the plaintext part to AES-GCM?

Yes. An alternative would be to add random bytes but I'd like to do something not complicated to implement that guarantees some kind of protection.

Would you make use of the Associated Data input?

Yes, if I recall correctly AES-GCM should provide that

Is the shared key static / negotiated for a "connection"?

The first version will make use of pre-shared keys, so no need to negotiate the key. For a future version I was thinking to get the RSA shared key creating a new protocol that will use RSA. All the applications will be configured by an administrator that will "install" the public keys of all the other applications.

Just to clarify, TLS seems too general-purpose and bulky to implement (OpenSSL seems really messy for instance), and I just wanted to add a basic encryption scheme to my application. What I'd like to know is if this design can satisfy that requisite and how good can be considered from a security point's of view.

[edit3] Totally open to suggestions if something similar already exists.

• Is this for a real (e.g., commercial) application or just for learning purposes? Jul 11 '18 at 12:34
• learning purposes but in the future who knows Jul 11 '18 at 12:38

The best way to learn something is to do it right. So I'd suggest just using TLS, but perhaps use a different library, like BearSSL. That way if it does turn into something more than a learning project, you don't have to redesign everything or risk being very insecure.

That said, here are some things to think about given your current path.

1. You should definitely protect the EncryptedProtocol struct using the associated data portion.
2. You need to make your protocol "future-proof". This means that if someone broke GCM or AES tomorrow, you should be able to easily adapt to needing to use new ciphers for modes. Thus, in your protocol header, you probably need a cipher version field. Maybe for now you only support AES-GCM, but that cipher version field will make it so you can change in the future if needed.
• Thank you for your reply. (1) can you clarify? is not ok to send that structure in plain? I read something about TLS and it seems there is a kind of header like that, with the length of the encrypted payload. (2) that's ok, in fact I included 2 bytes for the version (from the version of this protocol one can understand which cipher has been used). Jul 11 '18 at 13:04
• @tustu 1) means that AES-GCM has an input "associated data" and is suggested that the header for the encrypted data be fed into that input.
– SEJPM
Jul 11 '18 at 13:15
• @tustu, associated data is in plaintext, it is just included in the tag calculation, so an attacker cannot modify it without being caught. Jul 11 '18 at 13:43
• thank you for your explanations. so if I've understood well you're saying to add in the header an "associated data" field? I will continue reading about it though Jul 11 '18 at 14:03
• It's an application of the (very important) principle that you should never process any content that hasn't been integrity checked. IE check the MAC/authentication tag FIRST, and only if it is valid should you ever attempt to decrypt the message, parse the header, etc. Jul 11 '18 at 14:04