Trying to simplify my question, keeping only core concepts.
- Both user devices generates ECDHE key pairs. Send pub keys to each other. Generate shared secret.
- Device that requests secrets – presents dialog with the 6 digit code. 6 digit code is just first 6 bytes of SHA-512(shared_secret||A_identity||B_identity) and mod 10 of each byte.
- Device that expected to share secrets asks user to enter 6 digit code. And if code entered is correct (no man-in-the-middle + explicit user approve on action) – uses KDF on shared secret to generate transfer key and sends all data to other device.
As it's ephemeral keys and there is no ability to brute-force keys with the same fingerprint easily from first try - it is seems to be ok to use just 6 digit fingerprint. Server has no ability to intercept data or to introduce himself as legitimate device.
I'm not a security or crypto expert. Obviously I must not re-invent secure communication protocols. That's why I'm here to ask.
What are drawbacks of solution I propose? Are there any well known protocols for that case ?