If you want to be really sure, you could generate both the reply and the encryption keys from the challenge using a suitable key derivation function, such as HKDF (RFC 5869), e.g. as in:
salt = sender-ID | recipient-ID
PRK = HKDF-Extract(salt, challenge)
reply = HKDF-Expand(PRK, "reply", reply_len)
encrypt_key = HKDF-Expand(PRK, "encrypt", keylen)
Provided that HKDF is instantiated appropriately and that the underlying hash function is secure, it should not be possible for an attacker observing the reply to learn anything useful about the challenge or the encryption key, at least provided that the same challenge is never sent twice between the same parties. (If you want to eliminate even that minor loophole, include something unique, such as a timestamp, in the salt.)
Edit: Your originally proposed scheme could also be safe, but there are some potential pitfalls. For example, if you're using the same hash function to generate the reply as for HMAC, and if your challenge is longer than one hash input block, and you're not using any kind of salting to differentiate the two uses, then you could have a problem.
That's because, if the keystring is too long, HMAC will first hash it down to less than one input block using the underlying hash function, and then use the hash output as the actual key. Thus, anyone knowing the hash of the key would be able to compute the HMAC output just as easily as if they had the key itself.
That's one reason why I'd recommend using a KDF to generate both the reply and the encryption / HMAC keys, with different role strings as I suggest above. That's exactly what a KDF is designed for.