I have to send small (~10 bytes) authenticated messages over an insecure channel. I was thinking of an alternative to the "conventional" AES128-CTR-MAC, which will take 16 bytes for the CT and additional 2 or 4 bytes for the MAC. The motivation is that I'd like to reduce the message size as much as possible.
My alternative is to use a nonce and a counter, pass it to a HASH function, xor with the PT, and then AES encrypt. This would produce the CT.
Since PT is small, I was thinking off zero padding it (with two or 4 bytes). Then on decryption, I can compare the zero padding and use it as an authentication.
As I see it, this method is not prone to "bit switch" attack (as changing the CT will totally ruin the PT).
Do you think it's a reasonable way to do authentication? Is it enough to use a plain HASH function, or is it better to add a key to the HASH?
If, in addition, I also want to authenticate a non encrypted part of the message, It is OK to concatenate this message to the HASH input?
In addition: The transmitter and the receiver are time synchronized. and keep their internal (and equal) counter value. on each transmission, the transmitter and receiver increment their counter. The message is sent once every 10 seconds, and the receiver will accept only one packet on scheduled time. That way, the attacker can't brute force the MAC.