Designing a secure UDP-based communication protocol

Question

SUMMARY UPDATE:

I have changed the title of this question from "Using an encrypted packet counter as a counter value in AES-CTR" to "Designing a secure UDP-based communication protocol", because the discussion surrounding this question has evolved from my initial specific inquiry towards a more general overview of protocol design, thanks to the amazing answers provided here.

For anyone stumbling on this question when performing similar research, I will try to summarize the answers given and my key takeaways.

IMPORTANTLY: If you are considering designing your own protocol which is intended to be used in a real-world, production setting, please, evaluate EXISTING, standardized, and verified solutions and whether they can be used for your scenarios. My most important takeaway here is that properly designing a cryptographic scheme is very challenging, and unless you have a solid background in cryptography yourself, or someone can perform a thorough audit for you, you are very likely to miss something which will present a real-world vulnerability.

Some options which you can explore are TLS and DTLS, and especially DTLS if you need to layer a protocol on top of UDP and you cannot suffer the latency and head-of-line blocking issues incurred by TCP.

With all that said, if you are simply an enthusiast like me who would like to design a protocol for educational and/or hobbyist purposes, and you do not need incredibly strong security, I will try to summarize the main points and vulnerabilities you should consider:

• If using multiple encryption modes of operations for different purposes, you SHOULD use different secret keys for each. This also extends to other operations which use secret keys, such as HMACs. This is important, because using the same key everywhere is, generally, bad design, and can unintentionally leak information about the plaintext or worse.
• Use different, ephemeral keys for encryption whenever possible, even if they are derived from the same static key with the addition of some nonce. This ensures that even if encryption is broken for one session, then it will not compromise the security of other sessions, further complicating an attacker's job.
• Change encryption keys frequently even within one session. In particular, avoid using the same key with the same nonce more than once - as this invites a whole class of potential security issues. For more you can read RFC-5297 AES-SIV and RFC-8452 AES-GCM-SIV, as they offer resistance against nonce-misuse and nonce-reuse, outline the implications of such, and contain citations to further RFCs and papers describing the potential implications. For example, AES-GCM explicitly states that if nonces are reused then no security is offered. In addition, frequently changing encryption keys further guards the session against compromise since only a portion of the session will be broken if one key is revealed, instead of the entire session.
• 32-bit HMACs (or other signatures) are too small to provide sensible levels of security and expose a very real-world attack vector, making it feasible for an attacker to produce (even if by random chance) a packet that will be forged, but will seem authentic to the receiver. 64-bit is a minimum, 96-bit or 128-bit is recommended.
• Consider using AES-SIV, AES-GCM-SIV or some other AEAD instead of homebrewn HMAC + AES-CTR combinations, because AEADs already provide a tag for integrity and authenticity verification. They do impose significantly more overhead than a simple 32-bit HMAC, but that is the demand of stronger security.
• Ensure that your protocol is guarded against replay attacks, that is, when an attacker resends a packet that was already transmitted with the intention of re-triggering some action. There are several things to consider here:
• • An attacker might attempt to replay an extremely old packet that was sent before a packet counter wrapped around. To mitigate against this, ensure that keys and nonces are changed well before packet counter wrap-around occurs. In particular, the same header should never encrypt to the same ciphertext if counter wrap-around occured. Discard packets that cannot be verified using a recent key.
• • An attacker might attempt to replay a recent packet. To mitigate against this, ensure that recently received packets are tracked (for example, with a bitfield vector) and do not process a packet further if it was already received and processed. Discard packets that are too old, for example, if you know that all packets up to a certain packet counter value were received, immediately discard packets that are older than this value.
• Your protocol, in general, should give no indication that a packet failed to pass tag/HMAC verification. Prefer to silently drop these packets. Giving the attacker an indication of success/failure can give away unnecessary information about the encryption and provided plaintexts, and opens up venues for various oracle attacks.
• During the initial connection establishment, before symmetric encryption is established, make sure to verify everything and ensure you're not leaking any extra information. An attacker should not be able to alter any of the transmitted messages in a way that would allow them to impersonate either party, or subvert the connection establishment in a controllable manner (other than, maybe, mangling it entirely). When sending public keys, if possible, verify them using external means (certificates etc.) and use signatures. As mentioned above, the key used for the establishment of a shared secret should be ephemeral, but I think you might use a static key for the initial signatures and initial authentication.
• Research existing published attacks against popular protocols and encryption layers such as SSH, TLS, SSL and others, as these give valuable insight into the kinds of things you must definitely avoid.

This is all I can think of. If you have anything extra to add, please leave a comment, and I will edit this section correspondingly.

If you're still curious about designing your own protocol, I highly advise you to read the excellent answers by Ilmari Karonen, Gilles and Richie Frame, as they go in-depth on the various topics and outline flaws in my initial design, which you can see below. I hope that witnessing this process will be helpful to any aspiring hobbyists like me who stumble here :)

---

ORIGINAL QUESTION:

I am designing a networking protocol meant to be layered on top on UDP, where all data, including packet headers, is encrypted after initial public key exchange.

This protocol is not really meant to be used in a production setting yet, it is mostly for experimentation, fun, and challenging myself.

As part of the challenge I am trying to minimize packet header overhead without compromising security.

The current model I am considering looks something like this:

1. Endpoints exchange public keys (optionally verified via certificates or some other means) and calculate a shared key using ECDH (with Curve25519 or similar). They derive the actual key used for encryption using some kind of key-derivation function.
2. Each endpoint generates a nonce and sends it to the other endpoint. These nonces will be used for AES in CTR mode.
3. All packet data is encrypted, using AES in ECB and CTR modes:
   3.1. AES-ECB is used to encrypt packet headers, which include a packet counter (a 32-bit field starting from 0), a 32-bit HMAC derived from the entire packet (including header and payload) and other information related to the protocol operation.
   3.2. AES-CTR is used to encrypt the rest of the packet. The nonce generated at step 2 is added with the counter value multiplied by 128, and the resulting value is used as the IV for AES-CTR. Each subsequent 16-byte block is encrypted as usual, with the counter incremented for each next block. This gives me 2048 bytes of data that I can encrypt in a single packet, without the possibility of reusing the same effective nonce for different packets.
4. To decrypt the data:
   4.1 First the packet header is decrypted using AES-ECB, and the packet counter and the HMAC is extracted.
   4.2. We now know the IV for AES-CTR and can decrypt the rest of the packet.
   4.3. After we have the full packet contents in plaintext, we can verify that the HMAC is correct. If it is, we pass the packet up to the processing application.

I am wondering if this encryption scheme would have any obvious vulnerabilities if compared to something like AES-GCM.

From my understanding, the combination of the packet counter, various header fields, and the packet payload itself should produce a HMAC unique enough that it would prevent known-plaintext attacks against the encrypted header (i.e. the header ciphertext will always, or almost always, be different, even if the packet counter wraps around, since the HMAC is also different). And since the packet header is also encrypted, that further complicates extracting any data from it.

I considered using AES-GCM, but NIST recommends against using short (32 or 64 bit) tags, and I do not have a confident enough grip on the underlying cryptography to make an educated decision on whether I can satisfy all the necessary requirements to use AES-GCM securely. Since I am already using an HMAC for authentication and integrity validation, I am not sure if AES-GCM would give me any additional benefit over the proposed scheme anyway.

However, I am far from being a cryptography expert, and even though this scheme seems sound to me from the limited research I have conducted, I would like to hear some opinions from someone more educated than me.

---

EDIT (response to Ilmari Karonen's amazing answer):

Just to clarify, this protocol aims to be a flexible analog to SCTP, with support for ordered/unordered and reliable/unreliable packets and multiple independent streams that do not incur head-of-line blocking on each other.

One thing I'm concerned about is your use of a 32-bit authentication tag. Can you be sure that no attacker can afford to send four billion forged packets just to have one of them pass validation? If they might (and you may be surprised by how little time it takes to send four billion short packets at today's network speeds), then 32 bits is too little. Going up to even just 64 bits would give you a lot more security margin.

This isn't a point that I considered in my initial analysis, but it's a very valid one. Based on this, I will probably use a 64-bit HMAC for data-carrying packets, with an option to go up to 128 bits (perhaps configurable during connection negotiation), and a mandatory 128-bit HMAC on protocol control packets. I am still considering an optional downgrade to 32 bits for certain data channels where data is not very sensitive, and the application can decide if the extra 4 bytes are worth it.

I'm assuming that you're including the packet counter in the HMAC input, so that an attacker cannot change it and perform a replay attack. (Of course, you still need to explicitly check for duplicate and/or non-monotone message numbers to actually prevent replays.) You should also include some indicator of the packet's sender and its recipient in the HMAC input (even if you don't explicitly transmit such metadata with the packet!) to prevent an attacker from reflecting a message back to its sender and having it be accepted. On a two-party channel a single bit indicating whether the message is from Alice to Bob or from Bob to Alice would be sufficient.

Indeed. The way I planned to circumvent replay attacks further was to only accept packets that have not been received previously (for reliable ordered/unordered channels) and packets that are no older than a certain threshold (for unreliable unordered) channels. Unreliable ordered channels will, by their nature, only process packets if they are more recent than the last received packet.

That said, there are some things in your protocol that seem unnecessary, including the nonce exchange in step 2. You already have a packet counter that should serve as a perfectly good nonce, at least when combined with a single bit to indicate which of the two parties sent the message (assuming that the same key is used in both directions). Using a 32-bit nonce does mean that you can send at most 2^32 packets in either direction before you need to change keys, but depending on the intended use of the protocol, that might be sufficient. And, in any case, I don't see how your step 2 would help with that limitation (or anything else, really) in any way.

(There are ways to get around that limitation without increasing the number of header bits used for the packet counter, essentially by using a longer counter as the nonce but only transmitting the lowest 32 bits of it, and relying on the sequentiality of the packet numbers to let the receiver reconstruct the upper bits. Doing so in a way that won't let an attacker disrupt the secure channel by sending forged or replayed messages to desynchronize the counters is doable, as long as the message authentication is done right — in particular, the untransmitted upper bits of the counters must still be authenticated — but ensuring that all edge cases are accounted for can be a bit tricky.)

I want to reply to these points in a group, since I feel they are related somewhat.
My reasoning for choosing random nonces and transmitting them to the other side was:

1. It allows me to use a 128-bit nonce (as you recommended) while keeping the packet counter 32-bit. On counter wrap-around, the nonce itself is incremented by a value that is enough to ensure that the sum packet_counter + nonce is always unique. The other option I considered is to re-generate the nonce when the counter is about to wrap-around, and notify the other endpoint that the nonce is about to change.
2. From my understanding, since I use ECDH, then the shared key computed by both endpoints will be the same each time they establish a connection, assuming their ECDH keys do not change (which is desirable for authenticating the endpoints during connection establishment). The actual key derived from this shared key will thus, also, be the same each time. The nonces that are exchanged are used to prevent any data from being encrypted with the same keystream (which, as I understand, is a big no-no).
3. Since the nonces are asymmetric, this should also help with authentication in the case of reflecting a packet back to its sender - the sender will simply not decrypt the packet correctly since it will try to decrypt and verify it using the nonce of the other endpoint, and not its own nonce.

If a packet fails authentication, you also need to ensure that any response to it won't leak anything about the decrypted header (or the data!) beyond the fact that the packet was not valid. In particular, if an attacker could learn the decrypted packet header (or even a part of it), they could use this as an AES-ECB decryption oracle e.g. to confirm whether or not a guessed CTR plaintext block was correct or not.

I plan to silently discard any packets that fail authentication so as to not give away any information about what happened to the packet at all. I hope that since this protocol in typical use will also be a very chatty one, the attacker should not even be able to detect if an ACK was or was not sent to a packet they attempted to forge.

Also, if you're using the same AES key for both the ECB header encryption and the CTR data encryption, you need to ensure that no plaintext header can ever be a valid CTR counter block or vice versa. One way to do that would be to have some bit or group of bits within the 128-bit AES input block that is always all zero in the counter blocks, and never all zero in a valid plaintext header.

Of course, an alternative way to avoid these issues would be to generate two AES keys with the KDF: one for the ECB mode header encryption and the other for the CTR mode data encryption. That seems like the best option

I agree, using separate keys for ECB and CTR appears to be the most straightforward option, and I have no reason to insist on using the same key for both. Thanks for pointing that out. About the plaintext headers and CTR counter blocks - I am curious why that is an issue. Is this because a plaintext header will encrypt into the same ciphertext as some part of the keystream? That does seem fishy but I can't quite grok where the vulnerability arises from that.

As for the other suggestions, I will do my best to research all the things you suggested, and I greatly appreciate this in-depth analysis that you provided. It is very valuable.

Answer 1

At a glance, this looks mostly OK to me, at least assuming that I'm filling in the gaps in your description correctly and that there are no hidden security gotchas that aren't readily apparent from what you've written.

One thing I'm concerned about is your use of a 32-bit authentication tag. Can you be sure that no attacker can afford to send four billion forged packets just to have one of them pass validation? If they might (and you may be surprised by how little time it takes to send four billion short packets at today's network speeds), then 32 bits is too little. Going up to even just 64 bits would give you a lot more security margin.

I'm assuming that you're including the packet counter in the HMAC input, so that an attacker cannot change it and perform a replay attack. (Of course, you still need to explicitly check for duplicate and/or non-monotone message numbers to actually prevent replays.) You should also include some indicator of the packet's sender and its recipient in the HMAC input (even if you don't explicitly transmit such metadata with the packet!) to prevent an attacker from reflecting a message back to its sender and having it be accepted. On a two-party channel a single bit indicating whether the message is from Alice to Bob or from Bob to Alice would be sufficient.

Also, if you're using the same AES key for both the ECB header encryption and the CTR data encryption, you need to ensure that no plaintext header can ever be a valid CTR counter block or vice versa. One way to do that would be to have some bit or group of bits within the 128-bit AES input block that is always all zero in the counter blocks, and never all zero in a valid plaintext header.

If a packet fails authentication, you also need to ensure that any response to it won't leak anything about the decrypted header (or the data!) beyond the fact that the packet was not valid. In particular, if an attacker could learn the decrypted packet header (or even a part of it), they could use this as an AES-ECB decryption oracle e.g. to confirm whether or not a guessed CTR plaintext block was correct or not.

Of course, an alternative way to avoid these issues would be to generate two AES keys with the KDF: one for the ECB mode header encryption and the other for the CTR mode data encryption.

---

That said, there are some things in your protocol that seem unnecessary, including the nonce exchange in step 2. You already have a packet counter that should serve as a perfectly good nonce, at least when combined with a single bit to indicate which of the two parties sent the message (assuming that the same key is used in both directions). Using a 32-bit nonce does mean that you can send at most 2³² packets in either direction before you need to change keys, but depending on the intended use of the protocol, that might be sufficient. And, in any case, I don't see how your step 2 would help with that limitation (or anything else, really) in any way.

(There are ways to get around that limitation without increasing the number of header bits used for the packet counter, essentially by using a longer counter as the nonce but only transmitting the lowest 32 bits of it, and relying on the sequentiality of the packet numbers to let the receiver reconstruct the upper bits. Doing so in a way that won't let an attacker disrupt the secure channel by sending forged or replayed messages to desynchronize the counters is doable, as long as the message authentication is done right — in particular, the untransmitted upper bits of the counters must still be authenticated — but ensuring that all edge cases are accounted for can be a bit tricky.)

---

As a general footnote, I would strongly urge you to consider AES-SIV or AES-GCM-SIV. These modes do have slightly more packet length overhead than your proposed scheme (since you need to transmit the synthetic IV, which has to be considerably longer than 32 bits, in addition to your packet counter and other metadata), but if you can afford it, they provide better security properties and, being standardized, are less likely to harbor accidental security holes (at least if you use a well written off-the-shelf implementation, or carefully follow the standard when writing yours).

In particular, while SIV (and GCM-SIV) mode is defined with the assumption that nonces (and other "associated data") are transmitted in the plain (if at all), that's not actually necessary — e.g. prepending the nonce (and any other metadata) to the plaintext before SIV encryption will provide the same level of message authentication (since both the plaintext and the metadata get authenticated either way) and plaintext secrecy (prepending the nonce to the plaintext makes all plaintexts unique as long as nonces are not reused, which makes the DAE and MRAE security notions effectively coincide), while also keeping the nonces and metadata confidential (and, indeed, making the encrypted packets indistinguishable from random data).

---

It occurs to me that it might be possible to convert your scheme into an SIV-like DAE/MRAE secure scheme by basically using the encrypted header block as the initial CTR counter value for each packet. But verifying the security of such a scheme would require some non-trivial analysis (especially if you insist on using the same AES key for both header and data encryption; assuming two pseudo-independent AES keys ought to simplify things a lot) that I'm not ready to carry out right now.

Also, the short (H)MAC tag length seems likely to be a limiting factor here as well. My gut feeling is that 64 bits is the bare minimum likely to offer any reasonable security, and I'd feel a lot more comfortable with 96 bits (which would leave you only 32 bits for the nonce and any other metadata — although again you could always move some of that to a prefix of the plaintext, if needed).

---

Addendum: Let me answer some of your follow-up questions. (For the questions I don't explicitly answer below, the general answer is "yes, that looks more or less correct to me.")

From my understanding, since I use ECDH, then the shared key computed by both endpoints will be the same each time they establish a connection, assuming their ECDH keys do not change (which is desirable for authenticating the endpoints during connection establishment). The actual key derived from this shared key will thus, also, be the same each time.

That's not always, or even usually, true. While ECDH key exchange can be used like that (i.e. as a C(0e, 2s) scheme, as defined in NIST SP 800-56A rev. 3), it's more commonly used to establish a new ephemeral shared secret for each session based on (the participants' static keys, if any, and) random EC keys chosen by the participants at the start of the key exchange (i.e. as a C(2e, 2s) or a C(2e, 0s) scheme, depending on whether static keys are involved or not).

One important advantage of using ECDH this way is that it provides forward secrecy: once the session is over and the ephemeral shared secret, any keys derived from it and the random keys used to generate it have been discarded, any intercepted communication encrypted using keys derived from the ephemeral shared secret can no longer be decrypted even if the static keys of one or both participants are later compromised.

On counter wrap-around, the nonce itself is incremented by a value that is enough to ensure that the sum packet_counter + nonce is always unique.

OK, that looks reasonable, although AFAICT it doesn't really seem any different from my suggestion of using a longer packet counter and only sending its lowest 32 bits. Same thing, different way of looking at it.

Since the nonces are asymmetric, this should also help with authentication in the case of reflecting a packet back to its sender - the sender will simply not decrypt the packet correctly since it will try to decrypt and verify it using the nonce of the other endpoint, and not its own nonce.

That would indeed help, at least if you include the asymmetric "session nonce" as part of the HMAC input. But any other input that differs between the participants would serve that purpose just as well.

About the plaintext headers and CTR counter blocks - I am curious why that is an issue. Is this because a plaintext header will encrypt into the same ciphertext as some part of the keystream?

Yes. Whether this is actually exploitable or not depends on details of the encryption scheme and the attack model being considered. But if it's not proven not to be an issue, it's generally prudent to regard it as a potential one.

As a worst case scenario, if an attacker could somehow persuade on the the communicating parties to encrypt packets with chosen plaintext header blocks (and could then intercept the resulting ciphertext), they could use this AES-ECB encryption oracle to decrypt any and all AES-CTR encrypted data. Of course, this seems unlikely in your protocol, if only because your header blocks include the HMAC of the message data (and the attacker presumably won't know the HMAC key), but lesser variants of this attack might still be possible e.g. if the attacker could somehow learn the HMACs of some plaintexts.

Answer 2

Great answers have been given already, though when designing a protocol like this, there are certain attacks which will not occur, and the protocol is designed assuming they do not occur, but you extend or slightly modify something and now that attack breaks it wide up.

As such, based on comments and the other answers there are a few suggestions I would make:

The actual key derived from this shared key will thus, also, be the same each time

Since the nonces are asymmetric, this should also help with authentication in the case of reflecting a packet back to its sender - the sender will simply not decrypt the packet correctly since it will try to decrypt and verify it using the nonce of the other endpoint, and not its own nonce.

Storage is cheap, as are AES key changes. Derive more key material. Use one key for sending, the other for receiving, one for send auth, one for receive auth. A 512-bit hash can provide 4 128-bit keys. This keeps the nonces and counters for each data path independent, and you do not have to worry about asymmetry or synchronization. The key pairs between communicating endpoints will be flipped. If you need more keys, there are many KDFs which can pump out key material no problem, I like Keccak with bitrate equal to keysize for simplicity.

4.3. After we have the full packet contents in plaintext, we can verify that the HMAC is correct. If it is, we pass the packet up to the processing application.

You should HMAC your ciphertext, and only once it passes perform decryption.

This gives me 2048 bytes of data that I can encrypt in a single packet, without the possibility of reusing the same effective nonce for different packets

This is one of those design decisions that can come back to bite you hard, especially if you design the crypto around it. Give yourself a LOT more flexibility in the protocol for message size, so that if something changes in the future and you need larger messages, you are good to go. UDP packets can go up to 64KB, and maybe you change to something else that can be even larger, either way go big.

The other option I considered is to re-generate the nonce when the counter is about to wrap-around, and notify the other endpoint that the nonce is about to change.

You should have a key exchange WAY before something like that is about to occur, if a counter wrap occurs once a month, change keys every week. I am not sure how chatty your devices will be, but with 1 packet per second, a 32-bit counter wraps every 136 years.

Answer 3

Weakness: static DH

Endpoints exchange public keys (optionally verified via certificates or some other means) and calculate a shared key using ECDH (with Curve25519 or similar).

From my understanding, since I use ECDH, then the shared key computed by both endpoints will be the same each time they establish a connection, assuming their ECDH keys do not change (which is desirable for authenticating the endpoints during connection establishment). The actual key derived from this shared key will thus, also, be the same each time.

You rely on the establishment of the shared secret for authentication. This is possible, but a lot more fragile than relying on signatures.

Key establishment combines the private key with an input that is received from the network. An attacker can send malicious inputs in order to exploit weaknesses in this calculation. For example, failing to validate the peer's public key can leak the private key. Side channels can also leak the private key, and side channels are often easier to exploit when the attacker gets to choose the input.

With signatures, Alice uses her private key in combination with a message than she produces, and Bob uses his public key in combination with a message that he receives from the network. An attacker who wants to impersonate Alice has very limited ability to interfere with the signing process since the attacker can't control the input (the attacker may be able to submit some of the input, but that input gets hashed together with secret data, so the attacker has little control). The attacker can send messages to Bob, but since Bob is only using a public key, the verification process only needs to be functionally correct: oracles and side channels are not a concern.

Using ephemeral Diffie-Hellman is a lot safer than static DH. Since each connection uses a different private key, side channels are not a concern unless they allow extracting the key with a single observation, which is extremely rare. This does mean that you need a signature, which roughly doubles the computation time for key establishment.

Another advantage of ephemeral DH plus signatures as opposed to static ECDH is forward secrecy. With static keys, if an attacker logs traffic and eventually manages to crack either party's private key, they can decrypt all the traffic logs. With ephemeral keys, cracking one session's key only reveals the traffic of that session, and cracking the signature key allows the attacker to impersonate that party but not to decrypt past traffic.

Vulnerability: no defense against replay attack

Each endpoint generates a nonce and sends it to the other endpoint.

You need this nonce to avoid replay attacks. You should make sure to use different symmetric keys in each session, otherwise it's difficult to ensure that a packet from one session won't be valid in another session.

Looking at the subsequent description of the protocol, I don't see a defense against replay attacks. In particular:

3.1. AES-ECB is used to encrypt packet headers, which include a packet counter (a 32-bit field starting from 0), a 32-bit HMAC derived from the entire packet (including header and payload) and other information related to the protocol operation.

So the same message will be valid in all sessions.

A protocol should never generate the same ciphertext twice (except for simple resends), and the plaintext of each authenticated packet should contain some unique identifier that is never repeated with the same authentication key. Otherwise it's likely that the protocol allows some kind of replay attack.

Vulnerability: unauthenticated and non-fresh nonce

Each endpoint generates a nonce and sends it to the other endpoint. These nonces will be used for AES in CTR mode.

A man-in-the-middle can send a different nonce. I haven't worked out the consequences. It's probably not good, but there's a lot worse…

Even if you add a MAC to the nonce at this stage, this still allows the attacker to replay an old nonce. So the nonce is not actually a nonce.

Weakness: short HMAC

a 32-bit HMAC

It only takes 2 billion attempts to crack a 32-bit MAC on average. That's within the realm of plausibility for an online attack.

It's likely that uses of this protocol would be subject to collision attacks. Finding a collision, i.e. two packets with the same HMAC, only takes about 16 bits worth of effort (the birthday bound). The attacker doesn't get to determine what's in these packets, so the impact depends on the application protocol.

Weakness or limitation: ECB

AES-ECB is used to encrypt packet headers, which include a packet counter (a 32-bit field starting from 0), a 32-bit HMAC derived from the entire packet (including header and payload) and other information related to the protocol operation.

The other information must be exactly 64 bits. If you ever want to make the total amount of data larger, you'll have to use another mode, otherwise you'll run into the problem that ECB encrypts equal blocks of inputs as equal blocks of output.

Vulnerability: key reuse

You apparently use the same key for ECB and CTR. This is bad. Never use the same key for two different purposes. This is what broke OCB2, for example.

The same block gets encrypted with ECB and CTR if packet_counter + HMAC + info equals a CTR counter value. The CTR counter value is a public value (the nonce from step 2) which is constant within a session combined with the packet counter and the block index within the packet. Especially if the attacker can control the nonce, it's easy for the attacker to learn the encryption of certain blocks, and then recognize them when seen as the ECB-encrypted packet header.

Partial conclusion

I'll stop here. I'm not an expert on cryptanalysis by any means, and I've already poked a few holes. This is a pretty good effort, but far, far short of something that has a solid chance of being secure.

Use DTLS. It's been reviewed and validated by actual cryptographers.