Client login without sending a password

Question

I would love to have a system where the password is never send in plaintext (even with TLS) to authenticate a user. I've come up with this protocol, but I'd rather use something proven.

Registration:

1. Client (javascript in a browser) generates a 2048 bit RSA keypair (using Forge js)
2. Client generates hash from the users password using pbkdf2 with 68000 rounds
3. Client encrypts the private key with the above hash as the passphrase using AES 128 in OpenSSH format
4. Client sends the encrypted private key, public key, pbkdf2 salt and the username to the server

Login:

1. Client requests the encrypted private key and the pbkdf2 salt using the login name
2. Client generates hash from the users password using pbkdf2 with 68000 rounds and salt from the server
3. Client decrypts the private key using the pbkdf2 hash
4. Client requests a challenge from the server
5. Server sends an challenge that is encrypted using the user's public key using OAEP-SHA1 (Using cryptography.io)
6. Client decrypts the challenge and sends the plaintext challenge to the server
7. Server sends an limited use token to the client to use the api with

TLDR: Is there a login protocol where the client can prove authenticity to a server without sending a password. The user can then use a token for API calls that expire after a set time/a few calls. This system cannot use oauth.

Answer 1

I can see some weaknesses in your protocol. For example, it allows any attacker to request the encrypted private key and thus mount an offline dictionary or brute force attack on the password. (An incorrect password doesn't give you a properly formatted OpenSSH key.)

Thus this is a good idea:

I'd rather use something proven.

As mentioned in the comments SRP does exactly what you want. Here is description of the version 6(a) protocol.

To protect against compromised servers (like normal with password authentication), you'll want to incorporate your idea of using PBKDF2 (or another password hashing function). If you don't want the server to have to compute a slow password hash on each login attempt, you can use a normal hash as $H$ except when deriving $x$, which only the client needs to do. Or if you use an SRP library (as you should if possible) and it doesn't support this, you could calculate the $p$ you feed into SRP using PBKDF2 from the actual password.

Answer 2

You mention OpenSSH format, so may I assume you can use OpenSSH as part of your solution? If so, how about just using RSA key pairs with an authorized_key file?

Example

The server only ever has a public key for each user stored on it (nothing to lose to a server compromise) and the client just does RSA authentication (no passwords ever sent - client keeps control of their own, personal, private key).

Protocol is essentially, server encrypts challenge with public key, client decrypts with private key, and responds to challenge correctly, proving ownership of private key to server, and hence, identity.

You are wise to not want to implement a crypto protocol yourself from scratch, and use something tested. RSA keyless client login would seem to meet your requirements.

Once secured via this method, the server can provide whatever application-specific token it wants to the client over the ssh tunnel.