Without going deep into math, is there a way to make sure that non-open source programs like WhatsApp, FaceTime, Zoom, etc. are really using end-to-end encryption instead of just 'regular encryption' (i.e. the server has the cryptographic keys)?
Is there's a way to make sure that non-open source programs are really using end-to-end encryption?
Only by deep reverse-engineering. Which is hard, and might be illegal. Plus the apps are a moving target: they change like weekly. And, using end-to-end encryption is not proof that no interception is possible: this encryption (or its key generation) could be weak, accidentally, deliberately, perhaps surreptitiously. In the end, there's no practical way¹.
True story: I was the lead engineer for this² modem capable of end-to-end encryption with negotiation of session keys using Smart Cards, released in 1990, sold in France with some success. A director of our company insisted that we get approval from the authorities for these features, because that was raised in a conversation with Apple France, our main distributor. Soon after (mid 1991) I had a guy introducing himself with a military rank in my office, explaining that the law required that for crypto devices sold to the general public, there must be a way to find the key with a test requiring at most 240 steps. He required a detailed description of the protocol. I argued that changing it would break interoperability. The guy proposed addition of a side channel leaking the session key, so that if one end was using the rigged device, the session key would be available for one intercepting the line signal and knowing the method. I had the distinct impression this was standard procedure. In the end, I created a new packet type that older devices ignored, containing the session key RSA-encrypted, sent at moreless regular intervals, and new models did that. I remember the guy came for a demo. I insisted not to know the production private key, for my own safety. I still have the source code for that device, in 8051 assembly.
Note: the main feature of that device's point to point crypto was that, when you had made a set of Smart Cards, and inserted one in your modem, you could be confident that only someone holding another Smart Card from the set could connect. This feature was not compromised, as French law did not ask.
¹ No practical way to be confident that there is effective end-to-end encryption. It's sometime possible to prove that there is no end-to-end encryption. For example, if data goes thru a central server thru a https link protected using a server certificate, and the user messages are revealed by breaking that https encryption using a local proxy (a standard hack on one's device), then we are sure there is no end-to-end encryption.
² Scroll down and click on the wide picture or right arrow for more, including internals.
I have no connection with this website.
In the original question:
non-open source programs like WhatsApp, FaceTime, Zoom, etc
I think what you're really asking for is whether you can trust that something is using end-to-end encryption that you didn't personally install and host.
I don't think this is practical, even for open source software. The reason being that if you use someone else's service running jitsi or BigBlueButton, you're assuming they're running exactly what's provided on the public repository. Who really knows whether this is true.
So in my opinion, your best option is to find a service that uses some form of external verification (audit) that you trust, or run it yourself.
Agreed there is no way to be sure, in the sense that in general it's hard to be completely sure what is going on in any electronic device. Even if you dismantle it, and somehow observe what is going on, how do you know the device you are using to inspect it has not been hacked? Even supposing you built it yourself from scratch, how do you know someone didn't compromise it while you were asleep? You can take paranoia to almost any level.
In practice though, a reasonable test would be to test it externally, that is capture a sample of the packets of data exchanged and then examine the bits to see if the data is encrypted as advertised. Provided the external specification is published, or even if it is not, this should not be impossible. Of course this would only show that the captured packets were properly encrypted, it would be possible that the device sends un-encrypted packets on Sundays. In general, to be reasonably confident that cryptographic software does what it claims, you do need to examine the source code.