I'm currently working on a secure open source messaging system (https://github.com/DSMNET/DSMNET). Currently I'm using the cryptico.js (https://github.com/wwwtyro/cryptico) library to encrypt all the messages sent.
Now to authenticate user's my server will send a random string and the client will encrypt the string and sign it. The server will check if the signature matches the publickey stored for the user. User's will generate their public and private key with their password.