Basic encrypted custom protocol

Question

I've been suggested to ask here instead of stackoverflow, so here I am.

I'd like to encrypt the data of my application that uses a custom protocol I've created. I know about TLS and other protocols but I just want to do a basic encryption. The following code is C++, please ask me if you need any help to understand it.

My protocol is something like:

template<uint16_t thePayloadLength>
struct MyProtocolHeader{
uint16_t startOfFrame = 0xFFFF; // Protocol signature 
uint16_t version;
uint8_t src;
uint8_t dst;
uint16_t payloadLength = thePayloadLength;
uint8_t payload[thePayloadLength];
}; // the template part probably is going to disappear (too much code bloat)

My idea is to create two functions like:

template<uint16_t thePayloadLength
 /*EncryptedData*/ encrypt(const MyProtocolHeader<thePayloadLength>& plain);

 template<uint16_t thePayloadLength
MyProtocolHeader<thePayloadLength> decrypt(const /*EncryptedData*/& encryptedData);

that basically just do an encryption / decryption of my protocol struct. Then I can send the "EncryptedData" as a payload of an header like the following:

struct EncryptedProtocol{
uint16_t startOfFrame = 0xAAAA;
uint16_t version;
uint16_t sequenceNumber; // can be useful for UDP
uint16_t length; // lenght of the encrypted payload
};
// Encrypted payload here

I was thinking to use AES-GCM as encryption algorithm with the shared key already known by all the applications.

What do you think about it?

[EDIT] all the applications are outside Internet and they all know the public keys of all the other applications.

[EDIT2]

would you only feed the plaintext part to AES-GCM?

Yes. An alternative would be to add random bytes but I'd like to do something not complicated to implement that guarantees some kind of protection.

Would you make use of the Associated Data input?

Yes, if I recall correctly AES-GCM should provide that

Is the shared key static / negotiated for a "connection"?

The first version will make use of pre-shared keys, so no need to negotiate the key. For a future version I was thinking to get the RSA shared key creating a new protocol that will use RSA. All the applications will be configured by an administrator that will "install" the public keys of all the other applications.

Just to clarify, TLS seems too general-purpose and bulky to implement (OpenSSL seems really messy for instance), and I just wanted to add a basic encryption scheme to my application. What I'd like to know is if this design can satisfy that requisite and how good can be considered from a security point's of view.

[edit3] Totally open to suggestions if something similar already exists.

Answer 1

The best way to learn something is to do it right. So I'd suggest just using TLS, but perhaps use a different library, like BearSSL. That way if it does turn into something more than a learning project, you don't have to redesign everything or risk being very insecure.

That said, here are some things to think about given your current path.

1. You should definitely protect the EncryptedProtocol struct using the associated data portion.
2. You need to make your protocol "future-proof". This means that if someone broke GCM or AES tomorrow, you should be able to easily adapt to needing to use new ciphers for modes. Thus, in your protocol header, you probably need a cipher version field. Maybe for now you only support AES-GCM, but that cipher version field will make it so you can change in the future if needed.