Say I have an authentication protocol where the shared secret is never transmitted. The server passes a challenge to the client and the client calculates a response using an algorithm where the challenge and the secret is used.
When choosing the algorithm, what are the benefits and drawbacks of using either a hash (such as MD5) or a symmetric encryption algorithm (such as AES)?
You need to first understand the security properties you are trying to achieve. Possible security goals:
There may be other properties too.
Once you have a list of properties, you have to figure out why these properties hold. The reason should be things like "if the adversary could do that, they could also break the encryption" or "find collisions in a hash function."
How you choose to use a hash or an encryption function is based on the security properties you need and which primitive provides the correct notion of security. Hashing and encryption is not interchangeable. They provide very different properties.
There isn't anything we can say about the relative merits of hashing vs. encryption that applies universally to all challenge-response protocols. It is specific to the exact protocol and what properties you want from it. If you need more help, you could provide the two competing protocols and ask for help with (in a new question) evaluating their relative merits.
