First, check all the algorithms that you are using against official test vectors. This solves most of the problems, including the problem with the TLS 1.2 Client Finished Message you are describing. The problem in your case is not the approach, but the actual data being calculated in a chain of calculations. To figure out the exact step where it calculated wrong, compare test vectors and intermediate values (premaster secret, etc.) against other implementation of TLS, like OpenSSL.
In addition to the answer by dave_thompson_085, here is a practical advice on how to debug OpenSSL.
OpenSSL already implements a mechanism that allows you to trace various TLS internal values like Premaster Secret, Client Random, Server Random, Master Secret, Mac Key, Encryption Key, etc., by displaying these values to screen.
By default, the OpenSSL library is built with tracing disabled. To use the tracing functionality, it is therefore necessary to configure and build OpenSSL with the 'enable-trace' option. There is another build option, 'enable-ssl-trace', which adds "-trace" command line parameter for "openssl s_client" and "openssl s_server", but this option is not very useful when you are writing your own TLS implementation, because it only displays additional data for "s_client" and "s_server" rather than TLS internal values like key blocks.
When building OpenSSL, add the "enable-trace" parameter to "Configure". For example, for 64-bit Windows, call:
perl Configure enable-trace enable-ssl-trace VC-WIN64A
nmake
Then, the "openssl.exe" will read trace options from the OPENSSL_TRACE
environment variable. This variable accepts the coma-separated list of trace categories.
ALL
TRACE
INIT
TLS
TLS_CIPHER
CONF
ENGINE_TABLE
ENGINE_REF_COUNT
PKCS5V2
PKCS12_KEYGEN
PKCS12_DECRYPT
X509V3_POLICY
BN_CTX
STORE
To trace TLS internal values, you just need to specify the TLS
category to OPENSSL_TRACE
. For example:
set OPENSSL_TRACE=TLS
openssl.exe s_server
It will start accepting TLS connections on port 4433. Connect to it with your client and you will see on your console trace data like this:
TRACE[28:25:00:00]:TLS: Premaster Secret:
TRACE[28:25:00:00]:TLS: 0000 - 00 86 80 fa 69 b4 42 1d-e6 4b 20 b0 ca 42 6a 38 ....i.B..K ..Bj8
TRACE[28:25:00:00]:TLS: 0010 - f9 48 38 5b a1 1b e2 8e-6a 8f 4e 23 28 4b 9f 79 .H8[....j.N#(K.y
TRACE[28:25:00:00]:TLS: 0020 - ac 74 7d c3 4c 9d ba 0f-25 f2 97 89 3c ae 55 48 .t}.L...%...<.UH
TRACE[28:25:00:00]:TLS: 0030 - 92 df 77 4a 20 4e d8 a3-85 64 1e 35 55 72 c4 b5 ..wJ N...d.5Ur..
TRACE[28:25:00:00]:TLS: 0040 - a3 52 .R
TRACE[28:25:00:00]:TLS: Client Random:
TRACE[28:25:00:00]:TLS: 0000 - 5e 5c 2f 41 f4 8d 66 46-d7 fa 2d d2 16 2d 0d 31 ^\/A..fF..-..-.1
TRACE[28:25:00:00]:TLS: 0010 - de f5 5a 53 91 86 48 a7-ec af 9d 0c 16 03 d2 ec ..ZS..H.........
TRACE[28:25:00:00]:TLS: Server Random:
TRACE[28:25:00:00]:TLS: 0000 - 78 9c 50 33 93 38 70 f7-77 d6 43 5f be 20 4d 08 x.P3.8p.w.C_. M.
TRACE[28:25:00:00]:TLS: 0010 - c6 52 67 0d 8c 89 b0 0c-44 4f 57 4e 47 52 44 01 .Rg.....DOWNGRD.
TRACE[28:25:00:00]:TLS: Master Secret:
TRACE[28:25:00:00]:TLS: 0000 - 1f 5d bd 13 92 b9 34 ef-24 dc ee 2f 60 76 82 16 .]....4.$../`v..
TRACE[28:25:00:00]:TLS: 0010 - 0c a7 db 51 2a 69 46 23-8d 64 75 72 71 24 81 d1 ...Q*iF#.durq$..
TRACE[28:25:00:00]:TLS: 0020 - 52 3c 1a 68 9a d8 4a 00-38 5d 95 84 05 14 4f 69 R<.h..J.8]....Oi
TRACE[28:25:00:00]:TLS: dec 1
TRACE[28:25:00:00]:TLS: 0000 - 01 .
TRACE[28:25:00:00]:TLS: client random
TRACE[28:25:00:00]:TLS: 0000 - 5e 5c 2f 41 f4 8d 66 46-d7 fa 2d d2 16 2d 0d 31 ^\/A..fF..-..-.1
TRACE[28:25:00:00]:TLS: 0010 - de f5 5a 53 91 86 48 a7-ec af 9d 0c 16 03 d2 ec ..ZS..H.........
TRACE[28:25:00:00]:TLS: server random
TRACE[28:25:00:00]:TLS: 0000 - 78 9c 50 33 93 38 70 f7-77 d6 43 5f be 20 4d 08 x.P3.8p.w.C_. M.
TRACE[28:25:00:00]:TLS: 0010 - c6 52 67 0d 8c 89 b0 0c-44 4f 57 4e 47 52 44 01 .Rg.....DOWNGRD.
TRACE[28:25:00:00]:TLS: master key
TRACE[28:25:00:00]:TLS: 0000 - 1f 5d bd 13 92 b9 34 ef-24 dc ee 2f 60 76 82 16 .]....4.$../`v..
TRACE[28:25:00:00]:TLS: 0010 - 0c a7 db 51 2a 69 46 23-8d 64 75 72 71 24 81 d1 ...Q*iF#.durq$..
TRACE[28:25:00:00]:TLS: 0020 - 52 3c 1a 68 9a d8 4a 00-38 5d 95 84 05 14 4f 69 R<.h..J.8]....Oi
TRACE[28:25:00:00]:TLS: key block
TRACE[28:25:00:00]:TLS: 0000 - a0 0e 22 a5 56 fc 7a 20-88 0d 50 61 24 60 73 27 ..".V.z ..Pa$`s'
TRACE[28:25:00:00]:TLS: 0010 - 99 cb 03 48 db 27 12 80-7d d9 e6 bc 98 a9 90 e1 ...H.'..}.......
TRACE[28:25:00:00]:TLS: 0020 - 12 72 4d 76 04 c1 b6 d4-5b a8 98 e7 12 de 82 43 .rMv....[......C
TRACE[28:25:00:00]:TLS: 0030 - e7 21 bd 6c ba cc 65 1a-03 28 9a ac d4 00 0d 50 .!.l..e..(.....P
TRACE[28:25:00:00]:TLS: 0040 - 75 b3 bb 7f 6f f1 f7 28-5d cd cd d8 2f 8e 1b 21 u...o..(].../..!
TRACE[28:25:00:00]:TLS: 0050 - 8a fd bd 6b b6 3a dc 85- ...k.:..
TRACE[28:25:00:00]:TLS: which = 0021, mac key:
TRACE[28:25:00:00]:TLS: which = 0021, key:
TRACE[28:25:00:00]:TLS: 0000 - a0 0e 22 a5 56 fc 7a 20-88 0d 50 61 24 60 73 27 ..".V.z ..Pa$`s'
TRACE[28:25:00:00]:TLS: 0010 - 99 cb 03 48 db 27 12 80-7d d9 e6 bc 98 a9 90 e1 ...H.'..}.......
TRACE[28:25:00:00]:TLS: iv:
TRACE[28:25:00:00]:TLS: 0000 - 75 b3 bb 7f u...
TRACE[28:25:00:00]:TLS: dec 40
TRACE[28:25:00:00]:TLS: 0000 - 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
TRACE[28:25:00:00]:TLS: 0010 - 00 00 00 00 00 00 00 00-5d b0 62 5e 16 39 df 82 ........].b^.9..
TRACE[28:25:00:00]:TLS: 0020 - 02 9d d2 3d 19 7d 07 8f- ...=.}..