I'm looking for feedback on whether or not this is the proper way to approach password verification without transmitting the actual password. Are there any problems with this scheme.

We are working on a cloud based encrypted file system and have implemented the following steps to verify that the entered password is the same when entered a second time on a different machine. All implementations are from the Microsoft .NET Framework 4.0 (Client Profile). Also, all communication is SSL/HTTPS.

Initial activation for a file system

1. RNGCryptoServiceProvider is used to create two 128 bit random salts (s0, s1)
2. Rfc2898DeriveBytes is used with the entered password and s0 with 20,000 iterations to create the 32 byte key (k0)
3. s1 and k0 are appended to produce b0
4. SHA512Managed uses b0 to produce the hash (h0).
5. h0, s0 and s1 are encoded as base64 and sent to the server to be persisited.

1. s0 and s1 are sent to the client as base64 encoded strings
2. Both salts are decoded.
3. Rfc2898DeriveBytes is used with the entered password and s0 with 20,000 iterations to create the 32 byte key (k0)
4. s1 and k0 are appended to produce b0
5. SHA512Managed uses b0 to produce the hash (h0)
6. h0 is base64 encoded and sent to the server. If the hash matches the original hash now stored on the server the server responds with true, otherwise false ie: password does not match.

If all is good k0 is used to actually encrypt/decrypt the file system locally using AES-256.

-
only nitpick I see is that this forces all passwords to use the same salt –  ratchet freak Apr 25 '13 at 17:41
@ratchetfreak: Well the salts are only used for checksum verification. Password storage on each client is a separate process altogether. –  bic Apr 25 '13 at 18:09
Don't read more than the natural size from PBKDF2. Since Rfc2898DeriveBytes uses SHA-1, reading more than 20 bytes is a bad idea. It halves your KDF performance, without hindering the attacker. For client side password hashing I'd also crank up the iterations a bit to 100k or even higher. If that makes it too slow, replace .net's implementation with a fast implementation. Or switch to scrypt. –  CodesInChaos Apr 25 '13 at 18:21

The protocol you are trying to reinvent is called SRP.

Unless you have a compelling reason to believe that your protocol is secure, it should by default be assumed completely broken. Given that your protocol has seemingly no benefits over SRP, you should seriously consider going with something that has been designed and analyzed by cryptographers, has existing software implementations, and has been around for over a decade.

We are working on a cloud based encrypted file system...

Then you really should have somebody on staff who deeply understands cryptography. Your authentication protocol has a few sign that it was designed by novices. I don't mean to be rude or offensive, but cryptography is incredibly hard to get right. Thomas Ptacek said it best: if you think the dangerous stuff is in the primitives like the AES core, and if you just stick to the glue you'll be safe, you're gonna have a bad time.

4. s1 and k0 are appended to produce b0
5. SHA512Managed uses b0 to produce the hash (h0)

Why is this not an HMAC?

If all is good k0 is used to actually encrypt/decrypt the file system locally using AES-256.

What is the point of the authentication protocol, if a malicious client can simply opt not to bother sending $h_0$ to the server and attempt decryption anyway? If the sole purpose is to prevent accidental decryption with an incorrect key, then this is trivially solved by simply using an authenticated encryption mode. Which, really, you should be using anyway.

You've also walled yourself into a design that doesn't allow password changes without decrypting and re-encrypting everything. Instead, you could use a protocol where the client sends a password-protected secret key to the server. After authentication over something like SRP, the server sends a copy of the encrypted key back to the user. If they wish to change their password, they can simply re-encrypt and re-send the secret key.

Note: the above description is the general approach that I would try and look for in an existing protocol, and not a description of an existing, vetted protocol for this type of exchange.

-
Ok, great. Thanks for that insight. This is exactly why I posted the question in the first place. I absolutely don't want to reinvent the wheel as it were. Thanks again. –  bic Apr 25 '13 at 17:53
"it should by default be assumed completely broken". While I accept that, just for arguments sake, is the process described above in actuality broken? –  bic Apr 25 '13 at 18:02
Well, it relies on the CA system, and I think that's its main problem. $\:$ –  Ricky Demer Apr 25 '13 at 18:05
@RickyDemer: Sorry for my ignorance but what does "CA system" refer to in this context? –  bic Apr 25 '13 at 18:07