Expanding further on tylo's point regarding key management: when a user leaves the group, there is a need to create a new group key and distribute that to the remaining members.
The most basic way to do this is for the server to first establish a session key with each user via a Diffie-Hellman or RSA key exchange in a manner similar to TLS, then use each user's session key to encrypt the group key. In the event the group key needs to change, the server encrypts the new group key with each user's session key. Because n-1 rekey operations are required, where n is the number of users, this is not scalable in the context of large groups or frequent member changes. This can be mitigated with the use of auxiliary keys.
A common way on managing auxiliary keys is to arrange them in a tree structure, with the group key at the top and the users at the bottom, where each user holds the keys at its own node and all ancestor nodes. At the time a user joins the group, the server uses the user's session key to send it the auxiliary keys in addition to the group key.
In the case of a binary tree, a total of $2n-1$ keys are generated (one for each node), and each user holds $\log_2n+1$ keys. So when a user leaves the group, only $\log_2n$ keys need to be regenerated. This is described in more detail in this paper.
A more recent advancement involves using an m-ary tree where each level of the tree (other than the root, which is the group key) only contains m keys, and the total number of keys is $m\log_mn + n + 1$ (in this case, the individual session keys are not part of the tree). Besides each user only needing $\log_mn$ keys, it reduces the total number of rekeying operations in the event of multiple users leaving the group at once.