I have changed the title of this question from "Using an encrypted packet counter as a counter value in AES-CTR" to "Designing a secure UDP-based communication protocol", because the discussion surrounding this question has evolved from my initial specific inquiry towards a more general overview of protocol design, thanks to the amazing answers provided here.
For anyone stumbling on this question when performing similar research, I will try to summarize the answers given and my key takeaways.
IMPORTANTLY: If you are considering designing your own protocol which is intended to be used in a real-world, production setting, please, evaluate EXISTING, standardized, and verified solutions and whether they can be used for your scenarios. My most important takeaway here is that properly designing a cryptographic scheme is very challenging, and unless you have a solid background in cryptography yourself, or someone can perform a thorough audit for you, you are very likely to miss something which will present a real-world vulnerability.
Some options which you can explore are TLS and DTLS, and especially DTLS if you need to layer a protocol on top of UDP and you cannot suffer the latency and head-of-line blocking issues incurred by TCP.
With all that said, if you are simply an enthusiast like me who would like to design a protocol for educational and/or hobbyist purposes, and you do not need incredibly strong security, I will try to summarize the main points and vulnerabilities you should consider:
- If using multiple encryption modes of operations for different purposes, you SHOULD use different secret keys for each. This also extends to other operations which use secret keys, such as HMACs. This is important, because using the same key everywhere is, generally, bad design, and can unintentionally leak information about the plaintext or worse.
- Use different, ephemeral keys for encryption whenever possible, even if they are derived from the same static key with the addition of some nonce. This ensures that even if encryption is broken for one session, then it will not compromise the security of other sessions, further complicating an attacker's job.
- Change encryption keys frequently even within one session. In particular, avoid using the same key with the same nonce more than once - as this invites a whole class of potential security issues. For more you can read RFC-5297 AES-SIV and RFC-8452 AES-GCM-SIV, as they offer resistance against nonce-misuse and nonce-reuse, outline the implications of such, and contain citations to further RFCs and papers describing the potential implications. For example, AES-GCM explicitly states that if nonces are reused then no security is offered. In addition, frequently changing encryption keys further guards the session against compromise since only a portion of the session will be broken if one key is revealed, instead of the entire session.
- 32-bit HMACs (or other signatures) are too small to provide sensible levels of security and expose a very real-world attack vector, making it feasible for an attacker to produce (even if by random chance) a packet that will be forged, but will seem authentic to the receiver. 64-bit is a minimum, 96-bit or 128-bit is recommended.
- Consider using AES-SIV, AES-GCM-SIV or some other AEAD instead of homebrewn HMAC + AES-CTR combinations, because AEADs already provide a tag for integrity and authenticity verification. They do impose significantly more overhead than a simple 32-bit HMAC, but that is the demand of stronger security.
- Ensure that your protocol is guarded against replay attacks, that is, when an attacker resends a packet that was already transmitted with the intention of re-triggering some action. There are several things to consider here:
- An attacker might attempt to replay an extremely old packet that was sent before a packet counter wrapped around. To mitigate against this, ensure that keys and nonces are changed well before packet counter wrap-around occurs. In particular, the same header should never encrypt to the same ciphertext if counter wrap-around occured. Discard packets that cannot be verified using a recent key.
- An attacker might attempt to replay a recent packet. To mitigate against this, ensure that recently received packets are tracked (for example, with a bitfield vector) and do not process a packet further if it was already received and processed. Discard packets that are too old, for example, if you know that all packets up to a certain packet counter value were received, immediately discard packets that are older than this value.
- Your protocol, in general, should give no indication that a packet failed to pass tag/HMAC verification. Prefer to silently drop these packets. Giving the attacker an indication of success/failure can give away unnecessary information about the encryption and provided plaintexts, and opens up venues for various oracle attacks.
- During the initial connection establishment, before symmetric encryption is established, make sure to verify everything and ensure you're not leaking any extra information. An attacker should not be able to alter any of the transmitted messages in a way that would allow them to impersonate either party, or subvert the connection establishment in a controllable manner (other than, maybe, mangling it entirely). When sending public keys, if possible, verify them using external means (certificates etc.) and use signatures. As mentioned above, the key used for the establishment of a shared secret should be ephemeral, but I think you might use a static key for the initial signatures and initial authentication.
- Research existing published attacks against popular protocols and encryption layers such as SSH, TLS, SSL and others, as these give valuable insight into the kinds of things you must definitely avoid.
This is all I can think of. If you have anything extra to add, please leave a comment, and I will edit this section correspondingly.
If you're still curious about designing your own protocol, I highly advise you to read the excellent answers by Ilmari Karonen, Gilles and Richie Frame, as they go in-depth on the various topics and outline flaws in my initial design, which you can see below. I hope that witnessing this process will be helpful to any aspiring hobbyists like me who stumble here :)
I am designing a networking protocol meant to be layered on top on UDP, where all data, including packet headers, is encrypted after initial public key exchange.
This protocol is not really meant to be used in a production setting yet, it is mostly for experimentation, fun, and challenging myself.
As part of the challenge I am trying to minimize packet header overhead without compromising security.
The current model I am considering looks something like this:
- Endpoints exchange public keys (optionally verified via certificates or some other means) and calculate a shared key using ECDH (with Curve25519 or similar). They derive the actual key used for encryption using some kind of key-derivation function.
- Each endpoint generates a nonce and sends it to the other endpoint. These nonces will be used for AES in CTR mode.
- All packet data is encrypted, using AES in ECB and CTR modes:
3.1. AES-ECB is used to encrypt packet headers, which include a packet counter (a 32-bit field starting from 0), a 32-bit HMAC derived from the entire packet (including header and payload) and other information related to the protocol operation.
3.2. AES-CTR is used to encrypt the rest of the packet. The nonce generated at step 2 is added with the counter value multiplied by 128, and the resulting value is used as the IV for AES-CTR. Each subsequent 16-byte block is encrypted as usual, with the counter incremented for each next block. This gives me 2048 bytes of data that I can encrypt in a single packet, without the possibility of reusing the same effective nonce for different packets.
- To decrypt the data:
4.1 First the packet header is decrypted using AES-ECB, and the packet counter and the HMAC is extracted.
4.2. We now know the IV for AES-CTR and can decrypt the rest of the packet.
4.3. After we have the full packet contents in plaintext, we can verify that the HMAC is correct. If it is, we pass the packet up to the processing application.
I am wondering if this encryption scheme would have any obvious vulnerabilities if compared to something like AES-GCM.
From my understanding, the combination of the packet counter, various header fields, and the packet payload itself should produce a HMAC unique enough that it would prevent known-plaintext attacks against the encrypted header (i.e. the header ciphertext will always, or almost always, be different, even if the packet counter wraps around, since the HMAC is also different). And since the packet header is also encrypted, that further complicates extracting any data from it.
I considered using AES-GCM, but NIST recommends against using short (32 or 64 bit) tags, and I do not have a confident enough grip on the underlying cryptography to make an educated decision on whether I can satisfy all the necessary requirements to use AES-GCM securely. Since I am already using an HMAC for authentication and integrity validation, I am not sure if AES-GCM would give me any additional benefit over the proposed scheme anyway.
However, I am far from being a cryptography expert, and even though this scheme seems sound to me from the limited research I have conducted, I would like to hear some opinions from someone more educated than me.
EDIT (response to Ilmari Karonen's amazing answer):
Just to clarify, this protocol aims to be a flexible analog to SCTP, with support for ordered/unordered and reliable/unreliable packets and multiple independent streams that do not incur head-of-line blocking on each other.
One thing I'm concerned about is your use of a 32-bit authentication tag. Can you be sure that no attacker can afford to send four billion forged packets just to have one of them pass validation? If they might (and you may be surprised by how little time it takes to send four billion short packets at today's network speeds), then 32 bits is too little. Going up to even just 64 bits would give you a lot more security margin.
This isn't a point that I considered in my initial analysis, but it's a very valid one. Based on this, I will probably use a 64-bit HMAC for data-carrying packets, with an option to go up to 128 bits (perhaps configurable during connection negotiation), and a mandatory 128-bit HMAC on protocol control packets. I am still considering an optional downgrade to 32 bits for certain data channels where data is not very sensitive, and the application can decide if the extra 4 bytes are worth it.
I'm assuming that you're including the packet counter in the HMAC input, so that an attacker cannot change it and perform a replay attack. (Of course, you still need to explicitly check for duplicate and/or non-monotone message numbers to actually prevent replays.) You should also include some indicator of the packet's sender and its recipient in the HMAC input (even if you don't explicitly transmit such metadata with the packet!) to prevent an attacker from reflecting a message back to its sender and having it be accepted. On a two-party channel a single bit indicating whether the message is from Alice to Bob or from Bob to Alice would be sufficient.
Indeed. The way I planned to circumvent replay attacks further was to only accept packets that have not been received previously (for reliable ordered/unordered channels) and packets that are no older than a certain threshold (for unreliable unordered) channels. Unreliable ordered channels will, by their nature, only process packets if they are more recent than the last received packet.
That said, there are some things in your protocol that seem unnecessary, including the nonce exchange in step 2. You already have a packet counter that should serve as a perfectly good nonce, at least when combined with a single bit to indicate which of the two parties sent the message (assuming that the same key is used in both directions). Using a 32-bit nonce does mean that you can send at most 2^32 packets in either direction before you need to change keys, but depending on the intended use of the protocol, that might be sufficient. And, in any case, I don't see how your step 2 would help with that limitation (or anything else, really) in any way.
(There are ways to get around that limitation without increasing the number of header bits used for the packet counter, essentially by using a longer counter as the nonce but only transmitting the lowest 32 bits of it, and relying on the sequentiality of the packet numbers to let the receiver reconstruct the upper bits. Doing so in a way that won't let an attacker disrupt the secure channel by sending forged or replayed messages to desynchronize the counters is doable, as long as the message authentication is done right — in particular, the untransmitted upper bits of the counters must still be authenticated — but ensuring that all edge cases are accounted for can be a bit tricky.)
I want to reply to these points in a group, since I feel they are related somewhat.
My reasoning for choosing random nonces and transmitting them to the other side was:
- It allows me to use a 128-bit nonce (as you recommended) while keeping the packet counter 32-bit. On counter wrap-around, the nonce itself is incremented by a value that is enough to ensure that the sum
packet_counter + nonceis always unique. The other option I considered is to re-generate the nonce when the counter is about to wrap-around, and notify the other endpoint that the nonce is about to change.
- From my understanding, since I use ECDH, then the shared key computed by both endpoints will be the same each time they establish a connection, assuming their ECDH keys do not change (which is desirable for authenticating the endpoints during connection establishment). The actual key derived from this shared key will thus, also, be the same each time. The nonces that are exchanged are used to prevent any data from being encrypted with the same keystream (which, as I understand, is a big no-no).
- Since the nonces are asymmetric, this should also help with authentication in the case of reflecting a packet back to its sender - the sender will simply not decrypt the packet correctly since it will try to decrypt and verify it using the nonce of the other endpoint, and not its own nonce.
If a packet fails authentication, you also need to ensure that any response to it won't leak anything about the decrypted header (or the data!) beyond the fact that the packet was not valid. In particular, if an attacker could learn the decrypted packet header (or even a part of it), they could use this as an AES-ECB decryption oracle e.g. to confirm whether or not a guessed CTR plaintext block was correct or not.
I plan to silently discard any packets that fail authentication so as to not give away any information about what happened to the packet at all. I hope that since this protocol in typical use will also be a very chatty one, the attacker should not even be able to detect if an ACK was or was not sent to a packet they attempted to forge.
Also, if you're using the same AES key for both the ECB header encryption and the CTR data encryption, you need to ensure that no plaintext header can ever be a valid CTR counter block or vice versa. One way to do that would be to have some bit or group of bits within the 128-bit AES input block that is always all zero in the counter blocks, and never all zero in a valid plaintext header.
Of course, an alternative way to avoid these issues would be to generate two AES keys with the KDF: one for the ECB mode header encryption and the other for the CTR mode data encryption. That seems like the best option
I agree, using separate keys for ECB and CTR appears to be the most straightforward option, and I have no reason to insist on using the same key for both. Thanks for pointing that out. About the plaintext headers and CTR counter blocks - I am curious why that is an issue. Is this because a plaintext header will encrypt into the same ciphertext as some part of the keystream? That does seem fishy but I can't quite grok where the vulnerability arises from that.
As for the other suggestions, I will do my best to research all the things you suggested, and I greatly appreciate this in-depth analysis that you provided. It is very valuable.