You need to first understand the security properties you are trying to achieve. Possible security goals:
- It shouldn't be possible to answer convincingly if you don't know the secret
- If you do know the secret and provide a response, an eavesdropper shouldn't be able to extract the secret from seeing the challenge and response
- The challenger shouldn't be able to craft a challenge in such a way that it would leak information about the secret
- An eavesdropper shouldn't be able to replay responses later and have them accepted
There may be other properties too.
Once you have a list of properties, you have to figure out why these properties hold. The reason should be things like "if the adversary could do that, they could also break the encryption" or "find collisions in a hash function."
How you choose to use a hash or an encryption function is based on the security properties you need and which primitive provides the correct notion of security. Hashing and encryption is not interchangeable. They provide very different properties.
There isn't anything we can say about the relative merits of hashing vs. encryption that applies universally to all challenge-response protocols. It is specific to the exact protocol and what properties you want from it. If you need more help, you could provide the two competing protocols and ask for help with (in a new question) evaluating their relative merits.