I'm currently reverse engineering a program that uses Curve25519 key exchange in network communication. I have only a basic understanding of ECC, so maybe this thing just seems strange to me.
The client generates a private key, derives the public key from it, and sends it to the server. So far, that seems like a normal key exchange for me.
However, the server does not generate a simple public key and sends it to the client. The server generates a key pair and sends the public part to the client.
The client uses this public key and calculates a strange mechanism:
(Uses 25519ref10 - simplified pseudocode)
// The public key the server send to the client ge_p3 pubkey = ge_frombytes_negate_vartime(THE RECEIVED PUBLIC KEY FROM SERVER); // Scalarmult the sha512 with the public key from server. // This seems very strange to me... but its getting stranger. ge_p1p1 res = ge_scalarmult(THE SHA512 OF THE PUBLICKEY FROM THE SERVER + 8 EXTRA BYTES, pubkey); // Okay ge_p3 r = ge_p1p1_to_p3(res) // This is a hardcoded byte constant in the client binaries. // The private part of this can also be found in the server binaries. ge_p3 parkey = ge_frombytes_negate_vartime(A CONST PUBLICKEY); pargrp = ge_p3_to_cached(parkey); // Now this parkey const is getting added to the calculated scalarmult from hash & pubkey ge_p1p1 a = ge_add(r, pargrp); ge_p3 r2 = ge_p1p1_to_p3(a); // r2 will now converted to a byte array and the last byte is XOR'd with 0x80 result = ge_p3_tobytes(r2); result ^= 0x80;
This result is now used as the server public key.
So the server has to know the private key of "result" (and indeed, while this exchanging is happening, the private key of this calculation is in the memory of the server). But I cannot reverse this part of the calculation from the server binaries.
The only thing that I know, is that the public key is scalar (a hash of public key, public key) + const. Of course the server cannot calculate the private key from this special publickey. Does anyone have any idea how this private key could be generated?