Say I have an authentication protocol where the shared secret is never transmitted. The server passes a challenge to the client and the client calculates a response using an algorithm where the challenge and the secret is used.

When choosing the algorithm, what are the benefits and drawbacks of using either a hash (such as MD5) or a symmetric encryption algorithm (such as AES)?

  • 2
    $\begingroup$ Don't use MD5, it is broken (i.e. not collision-resistant any more). Even if it might be secure in your protocol, it is better for your reputation to use SHA-2 instead. $\endgroup$ Nov 30, 2011 at 9:39

1 Answer 1


You need to first understand the security properties you are trying to achieve. Possible security goals:

  • It shouldn't be possible to answer convincingly if you don't know the secret
  • If you do know the secret and provide a response, an eavesdropper shouldn't be able to extract the secret from seeing the challenge and response
  • The challenger shouldn't be able to craft a challenge in such a way that it would leak information about the secret
  • An eavesdropper shouldn't be able to replay responses later and have them accepted

There may be other properties too.

Once you have a list of properties, you have to figure out why these properties hold. The reason should be things like "if the adversary could do that, they could also break the encryption" or "find collisions in a hash function."

How you choose to use a hash or an encryption function is based on the security properties you need and which primitive provides the correct notion of security. Hashing and encryption is not interchangeable. They provide very different properties.

There isn't anything we can say about the relative merits of hashing vs. encryption that applies universally to all challenge-response protocols. It is specific to the exact protocol and what properties you want from it. If you need more help, you could provide the two competing protocols and ask for help with (in a new question) evaluating their relative merits.

  • $\begingroup$ Thanks. Let's hope someone can explain the relative merits as they pertain to the properties you listed, as that's exactly what I'm asking for. I don't have a specific protocol, this is just for learning. $\endgroup$
    – Niklas
    Dec 1, 2011 at 8:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.