While reading about Triple Diffie-Hellman, I was wondering how it can prevent MITM if the server decides to spoof key sets.
For example, Alice and Bob are connected using a server.
Alice ----- Server ----- Bob
What if the server, with malicious intention or under government request, tries to spoof the key sets? Is it possible to compromise the communication in such cases even using X3DH?
I'm assuming here that you are talking about this Signal defined protocol.
It depends. It basically hangs on one of the very first security consideration:
- Security considerations 4.1. Authentication
Before or after an X3DH key agreement, the parties may compare their identity public keys IKA and IKB through some authenticated channel. For example, they may compare public key fingerprints manually, or by scanning a QR code. Methods for doing this are outside the scope of this document.
If authentication is not performed, the parties receive no cryptographic guarantee as to who they are communicating with.
Now if authentication is not performed for one of the parties then the protocol would allow the impersonation of that party by controlling (or acting as, if this is not protected otherwise) the server. That's not entirely the same as a MiTM attack - only the untrusted party can be impersonated after all.
Only if the public keys of both parties are not trusted can a full MitM attack be performed. An MitM means that both parties are impersonated, after all.
Note by the way that in the rather dumbed-down description of the protocol it seems that Alice is not authenticated. That's not true.
See the following line in the protocol:
$$\mathrm{DH1 = DH(IK_A, SPK_B)}$$
Here $\mathrm{IK_A}$ must clearly be the private key of Alice and $\mathrm{SPK_B}$ must be the public signed prekey of Bob. That's a horrible notation that they use because $\mathrm{IK_A}$ is introduced earlier as a public key and $\mathrm{SP}$ is commonly used to refer to a secret (or private) key. This is all done under the because:
All public keys have a corresponding private key, but to simplify description we will focus on the public keys.
and that led unfortunately to the invalid description of the DH protocol.
Still, that does mean that Alice is authenticated as well, not by using a signature, but by using her authentication key pair as part of the calculation of the shared secret key.
Bob of course needs to use the corresponding public key in his calculations, but that is hidden by the following line:
Using these keys, Bob repeats the DH and KDF calculations from the previous section to derive SK, and then deletes the DH values.
This means that Alice identity is confirmed implicitly by verifying the authenticated, encrypted part of the initial message. It's kind of scary that this part about Alice's authentication is missing entirely from the protocol description; hopefully implementations will handle bad authentication tags within the ciphertext of the initial message as authentication failures.
