Here is setup:

There are two devices with the app of the same user. They have already been authenticated by the server using sms code. One device have pair of x2559 keys that we need to share securely to other device. Server is not trusted and can perform man-in-the-middle attack. Moreover it can simulate other device to get keys from the master device. We don't want to use thirdparty communication channel, don't want to export keys to files etc.

I have not found exact solution to the case, but number of look-a-like cases. Most appealing implementation (in the matter of UX) is what I have seen on Apple devices with confirmation dialogs with random 6 digit keys.

Proposed solution:

1. Both devices generates ECDHE key pairs. Send pub keys to each other. Generate shared secret.
2. Device that requests secrets – presents dialog with the 6 digit code. 6 digit code is just first 6 bytes of SHA-512(pubKeyA||pubKeyB||A_identity||B_identity) and mod 10 of each byte.
3. Device that expected to share secrets asks user to enter 6 digit code. And if code entered is correct (no man-in-the-middle + explicit user approve on share action) – uses KDF on ECDHE shared secret to generate transfer key and sends all secrets to other device.

As it's ephemeral keys and there is no ability to brute-force keys with the same fingerprint easily from first try - it is seems to be ok to use just 6 digit fingerprint.

I'm not a security or crypto expert. Obviously I must not re-invent secure communication protocols. That's why I'm here to ask.

From the view of UX – I do want to use N-digit code authentication. But what's the right way to do it ? Well known protocols for authenticated one time DH. What are drawbacks of solution I propose?

Trying to simplify my question, keeping only core concepts.

Proposed solution:

1. Both user devices generates ECDHE key pairs. Send pub keys to each other. Generate shared secret.
2. Device that requests secrets – presents dialog with the 6 digit code. 6 digit code is just first 6 bytes of SHA-512(shared_secret||A_identity||B_identity) and mod 10 of each byte.
3. Device that expected to share secrets asks user to enter 6 digit code. And if code entered is correct (no man-in-the-middle + explicit user approve on action) – uses KDF on shared secret to generate transfer key and sends all data to other device.

As it's ephemeral keys and there is no ability to brute-force keys with the same fingerprint easily from first try - it is seems to be ok to use just 6 digit fingerprint. Server has no ability to intercept data or to introduce himself as legitimate device.

I'm not a security or crypto expert. Obviously I must not re-invent secure communication protocols. That's why I'm here to ask.

What are drawbacks of solution I propose? Are there any well known protocols for that case ?