I'm analyzing some software and finding key exchanging procedure. Its looks like very similar with Diffie–Hellman, but not the same. I can’t understand the purpose of this.
The step of algorithm is:
- Generate random KEY (128 byte)
- Get constant G1 and P1
- Calculate Y1 = G1 ^ KEY mod P1 // looks like Diffie–Hellman
- Get another const G2 and P2 // ???
- Calculate Y2 = Y1 ^ G2 mod P2, so Y2 = (G1 ^ KEY mod P1) ^ G2 mod P2
- Transmit Y2 and receive key from other side FY2
- Calculate Y3 = FY2 ^ KEY mod P1 // private key
- On next step Y3 using as seed for stream cipher
I trying to simulate this procedure to get key exchange and its fail. Y3 keys on both side are not same. Looks like procedure on other side is different from this.
Any suggestion how can this algorithm works?