For participants joining the call, the broadcaster would have to send the session key encrypted with their and joining user's long lived shared key. Which is simple enough.
When a user leaves the call, if the session key is unchanged, they can still decrypt the stream. The only secure option is change the session key and rebroadcast encrypted copies of it to all remaining participants. This ensures that when a user leaves, they can't continue to listen in.
Less obviously, when a user enters, if the key is not changed, the key they are sent decrypts data from before they joined.
The best solution is to rekey anytime the set of authorized users changes.