I wonder what is wrong with this scheme for authentication.
- Server sends random challenge, C
- Client returns (C, B(P)) encrypted in H(B(P)) where B(P) is high work factor salted hash of the user passphrase (e.g., bcrypt), and H() is a low work factor hash
- Server has K = H(B(P)) in database, and can use it to decrypt (C,B(P)) and verify that H(B(P)) = K and that C is equal to the challenge
I wonder is B(P) adequately protected by encryption in key = H(B(P)) when sent over the network? Or, other problems?
EDIT: Let me clarify based on comments. I obscured the main question by splitting the password hash function into 2 stages. I think the answer to my question is the same if there is no B(P) and H() has high work factor. So, client is submitting (C,passphrase) encrypted in H(passphrase). Server knows H(passphrase), as in typical password authentication, so can decrypt. Then can compute H(passphrase) to verify client knows passphrase. It is like classic bcrypt based authentication, but using H(passphrase), rather than SSL, to secure the transport. My concern is how to safely do encryption when the key is derived from the data being encrypted.
The reason for the 2 stage H(B(P)) in my original question is to minimize load on the server, which is the main aim of the protocol, to allow run of the mill hardware to separate wheat from chaff in face of gbps DoS attacks.
EDIT2: Also, I left out how the server proves to the client that it also knows K, which is necessary to prevent active man in the middle. In step 2 the client includes a random session key encrypted in K, which is used to compute a message authentication code on all subsequent messages, including the server return.