I'm learning various cryptography protocols and use cases and since I'm developing a USB device and PC application, it would be nice to use encrypted communication.

I studied TLS 1.2 key exchange but mostly focuses on the one-way authentication only (client authenticates server), althought mTLS exists too. Aim is to:

  • Make sure device only works with proprietary PC software
  • Make sure PC software does not work with any other device but the one designed by myself (cloning)

I have these means available:

  • RootCA, available on the PC tool and on each device - identical for all.
  • Device specific private key (later called static private key) and its corresponding certificate, signed by RootCA. Every device has its own pair. These are securely loaded in the device
  • PC tool has its own private key (later called static private key) and certificate, signed by RootCA

Cryptographic means used:

  • SHA256 for hash function
  • AES-256-CBC for symmetric encryption
  • ECC-256 for PKI (secp256r1)
  • ECDH for shared pre-master generation
  • HKDF key derivation function for other keys generation
  • Cryptographically certified RNG on device side, win32 API RNG on PC side

There are 4-5 steps process for key exchange, quickly:

  • Step1: Host connects to device via USB, generates cryptographically secure random and sends it as a challenge
  • Step2: Device:
    • generates its own random
    • generates ephemeral public/private keys pairs
    • double hashes (host_random||device_random||device_eph_pubkey) and signs it with its static private key.
    • Sends back all data + its own certificate
  • Step3: Host can now check authenticity of a device (if cloned or genuine). If all is OK, it:
    • generates its own ephemeral keypair
    • double hashes (host_random||device_random||host_eph_pubkey) and signs it with its own static private key
    • generates 256-bit AES symmetric key with ECDH as it has all the data available, using its own ephemeral private key and client's public key. This is a pre-master, that is used to derive 2 more key with HKDF, one for host write and another for client write
    • AES-CBC-256 encrypts (host_random||client_random) with host write key using AES-256 CBC mode
    • sends all data back, including its own certificate
  • Step4: Client authenticates host by checking certificate (using rootCA) and signature with the received certificate. If everything is OK, it:
    • generates pre-master 256-bit symmetric key from host public and device private + 2 more for client write and host write.
    • tries to decrypt received message with symmetric key and checks if it matches (host_random||client_random)
    • AES-CBC-256 encrypts the (host_random||client_random) with client write key and sends it back
    • puts itself in the I'm ready to communicate with encrypted connection from now on mode
  • Step5: Host decrypts received message with client write key and checks if it matches (host_random||client_random)
    • If result is OK, then it is ready to communicate over encrypted channel

I'm however not sure about the symmetric encryption and data checks. I'm simply encrypting (host_random||client_random) on one side and expect to get the same (after decrypting) on other side. Theoretically, if data are tampered, decryption should yield wrong result on decryption side, which will not match the expected (host_random||client_random)

  • Is this step OK as it is, or shall I use any other method (HMAC) for symmetric encryption checks?
  • Is protocol OK or is there any obvious vulnerability?
    • I use random to prevent replay attack
    • I use certificates to prevent MITM attack
    • I use ephemeral keys in step3 to prevent replay attack of the original request
    • I use double hash to prevent length extension attack. It is not applicable here, but it can be a good practice
    • I use ephemeral keys to rotate symmetric key and confuse data between various sessions. This should enable forward secrecy
  • Could I do some step3 activity in step1 instead and still not be vulnerable to replay attack?
    • generation of ephemeral keys
    • generation of host random
    • generation of signature of hash of above
    • all sent to device + certificate proving signature
    • This can be replicated later, especially if client will use this info as authentication, so I believe answer is no

Step by step TLS exchange between PC application and USB device

  • $\begingroup$ If you embed any sort of credentials in software you distribute to others, people will be able to extract it and use it. That cannot be an effective control, so I don't believe there's any effective way to achieve your two goals. $\endgroup$
    – bk2204
    Jan 28 at 23:07
  • $\begingroup$ @bk2204 sorry, I may have misunderstood you. You mean there is no better way to do key exchange with authentication, than the one presented above, or is even this one problematic? $\endgroup$ Jan 29 at 6:57
  • $\begingroup$ I mean your PC tool and private key aren't private because they can be trivially extracted, so they provide no security. Thus, your device cannot work only with your proprietary software. Anyone can reimplement the protocol and take the key without negative consequence, and moreover, this is explicitly allowed by law in many countries (including the U.S. and Canada) as "reverse engineering for compatibility". $\endgroup$
    – bk2204
    Jan 29 at 22:20
  • $\begingroup$ OK, fair enough from the sense that anyone can extract private key from .exe file and/or change it. If we assume PC is another device with securely stored private key, is last step (AES encryption test) sufficient to detect no tampering, or do we need HMAC? $\endgroup$ Jan 29 at 22:27


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.