2
$\begingroup$

I've been trying to manually decrypt some TLS 1.3 traffic for educational reasons, and have stumbled across a roadblock. So far, I've been able to complete most of the key schedule, including deriving the correct handshake traffic secrets from the ECDHE keys. However, I haven't been able to find much information about how the ciphertext and authentication tag are formatted within Application Data messages.

Here is the full record.

1703030017bf687d10e2f209661418d92aaf3626dfe5670f3127d6ed

And here is the server handshake traffic secret:

475a0f3b5a86d4797e24545c2eb6388de9507e5e0bc367c74c3e78e0d8033f34b721a2e3a1d89ec5990287954402fc43

Wireshark tells me that decrypting this message yields the plaintext 08 00 00 02 00 00, corresponding to the Encrypted Extensions message.

From what I can tell, the "associated data" should be the bytes 17 03 03 00 17, and the rest of record is ciphertext + authentication tag. RFC 8446 says that the TLS_AES_256_GCM_SHA384 cipher suite is defined in RFC 5116, but RFC 5116 says:

This document does not specify any particular encoding for the AEAD inputs and outputs, since the encoding does not affect the security services provided by an AEAD algorithm.

Yet all my attempts to decrypt the data have failed to authenticate, which leads me to two possibilities:

  • I am deriving the traffic keys wrong. I know that my HKDF-Expand-Label routine works since all other secrets are derived correctly and match the values logged by OpenSSL. However, admittedly I'm not 100% sure what the correct key and IV length are. Currently I am using a 32-byte key and 12-byte IV. I don't know if that's what TLS 1.3 uses.
  • I am reading the authentication tag/ciphertext wrong. I haven't been able to find any references on what the proper format is, so all of my attempts in this area have just been guesses. In particular, I'm not sure how long the authentication tag is for this cipher suite, though googling suggests that it's 12 bytes.

What am I doing wrong?

Improve this question
$\endgroup$
2
  • $\begingroup$ I just realized that I completely glossed over the part of the spec mentioning that the IV is XOR'd by the record sequence number.. problem not solved, but now tantalizingly close. $\endgroup$
    – adrian
    Commented Feb 21, 2022 at 7:21
  • 1
    $\begingroup$ 5116 defines only the AEAD primitives, usable in all kinds of crypto applications, including but not specific to TLS ciphersuite(s). Format and building of the encrypted record is in 8446 5.2, and nonce generation (which you seem to have found) in 5.3. $\endgroup$
    – dave_thompson_085
    Commented Feb 21, 2022 at 23:40

1 Answer 1

Reset to default
2
$\begingroup$

It turns out I was making not one but two mistakes!

  • I did not read the section of the RFC explaining that the per-record nonce was calculated by XORing the sequence number against the IV.
  • I was using an incorrect tag length of 12 instead of 16.
Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Could you please post the full solution for encrypt/decrypt? $\endgroup$
    – xpepermint
    Commented Jun 11, 2022 at 5:53
  • 1
    $\begingroup$ @xpepermint here's my code: GitHub $\endgroup$
    – adrian
    Commented Jun 11, 2022 at 18:00
  • $\begingroup$ @adrian could you please republish the final GitHub repo? thank you $\endgroup$
    – miran80
    Commented May 8, 2023 at 18:47
  • 1
    $\begingroup$ bah, link rot! it's now here. $\endgroup$
    – adrian
    Commented May 9, 2023 at 5:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.