I'm working on a chat system that will be end-to-end encrypted, and I've run into a problem. As you know, in end-to-end encrypted chats, the server isn't trusted at all; it's why we have fingerprint checking etc.
The problem came about when I was considering group chats. If Alice, Bob, and Eve are communicating, and Jill decides to join the server, we'll have to re-do the key exchange for her. But what if Jill is a malicious attacker? We'll be giving her the encryption keys! The solution I had in mind was to place each
"trustworthy" client's public key in a file on the server, just like SSH does, and then the clients can identify themselves with a digital signature on connection. Now, as D.W. pointed out, this requires trusting the server, which is against standard end-to-end spec. In fact, any solution that allows the server to choose who is in the chat requires trust in the server.
Because what if the server is compromised or dishonest, and instead of attempting an MITM attack, it decides to add his own public key to the authorised_keys file? Then he can connect as a legitimate user and steal the session key. (This is how SSH does it, and I'm not sure how it mitigates this possibility, so as a side-question, clearing this up would be helpful.)
In an end-to-end encrypted system, what degree of control does the server usually have? I mean Signal Messenger has to have thought about this scenario and mitigated it.
Let me direct my question like so: what is the recommended practice for deciding who is a part of an end-to-end encrypted group chat? By recommended practice, I mean to communicate that an answer that draws from a well known and trusted protocol, or thing, like the signal protocol, would be optimal. The requirements are that the server should not be trusted and confidentiality and security for the clients should hold up in the adversarial model. They should also hold up if the server is compromised.
Obviously if the server cannot decide who is in the group, then the clients would somehow have to authorise each additional member, but seamlessly. How should I handle this? The idea is that the server is always up and clients can join and disconnect at any time, like a chat room.