We think that the player will not be able to get this key by extracting it from the memory or somehow else.
Forgive me, but I'm skeptical. If you really have figured out a way to do this --- and plenty of well-funded, intelligent people have tried and failed --- I'd recommend slapping a patent on it and making millions off of licensing fees.
How safe is this implementation?
Regardless of what you end up doing, the key will probably be the weak link. Or perhaps an attacker will try to read the decrypted packets from memory instead of the key.
I'm assuming that when you say "A fast hash function with a key will be used for authentication" what you mean is that you'll be using HMAC for authentication. If you mean something else, please clarify.
Using HMAC and CTR is a sound approach. Just make sure to authenticate the ciphertext ("encrypt-then-MAC"), rather than the plaintext ("MAC-then-encrypt" or "MAC and encrypt").
The real devil is in the details, though. You've given a broad overview of your approach, not a specific algorithm/protocol, and not an implementation. Each of these two transitions has the potential to go awry. For example, will your protocol prevent replay attacks? Will your implementation generate CTR nonces securely?
4-6 bytes (not 16) will be used for the tag.
I wouldn't use a tag this short to protect financial data, but for a game, it's probably fine.
Which hash function is better to use?
This depends on your definition of "better". SHA-2 will probably be the most secure. SHA-1 will probably be secure enough. HMAC-MD5 hasn't been broken, but using it is generally not recommended since MD5 has been broken. On the other hand, if your definition of "better" includes performance, than using MD5 may be worth the security risk (considering the relatively low stakes of the information you're protecting).
Can we use the same key for both the hash function and AES?
Do not do this. This may not lead directly to any obvious attacks, but key reuse in general will anger Crypto Gods because it violates a key assumption (heh) that cryptographers make when designing these algorithms. It's the crypto equivalent of not doing any bounds checking on your buffers. (And if you later switch to using something like CBC-MAC for authentication, key resuse will be come a definite problem.)
If transmitting two keys is problematic for whatever reason, you can securely derive keys from a master key by doing something like this:
encryption_key = HMAC-SHA1(master_key, "Encryption key")
authentication_key = HMAC-SHA1(master_key, "Authentication key")
A couple suggestions.
This wasn't part of your questions, but I'd recommend using a dedicated authenticated encryption scheme such as GCM (galois counter mode) instead of separate MAC and encryption algorithms (such as HMAC-SHA1 and CTR-AES). Doing so will prevent some common implementation errors while quite possibly increasing performance.
Regardless of whether you use HMAC or GCM, look into using "associated data" (sometimes called "additional authenticated data"). The idea here is that if there's data that doesn't need to be encrypted (i.e., kept secret from the players) but still needs to be authenticated (i.e., the players shouldn't be allowed to tamper with it), there's no need to eat the performance cost of encrypting it. A crypto library that offers GCM will likely take an optional argument containing a string of associated data.
