Can we do better?
Yes. See: On Ends-to-Ends Encryption: Asynchronous Group Messaging with Strong Security Guarantees.
To support significantly larger groups?
Our results demonstrate that ART is practical for reasonably-sized groups, with key tree setup and message sending both taking a few milliseconds for groups of size ten and on the order of one second for groups of size 1000.
While maintaining end to end encryption? Will we be able to still maintain other properties, like forward secrecy?
The important property that we care about here is Post-Compromise Security (PCS), defined as:
Moreover, some modern messaging protocols offer a property called Post-Compromise Security (PCS), sometimes referred to as “future secrecy” or “self-healing”. For PCS, even after Alice’s device is entirely compromised by an adversary, she may later be able to establish secure communications with others after a single unintercepted exchange.
These trees establish new group shared keys when any member joins or leaves the group, or when each member decides to rotate their key. In double-ratchet keys usually rotate after the first message from each user since the key changed, although some implementations, specifically Matrix' Olm, opt to be less noisy; re-keying every N-messages or after T-seconds.