generate hash of all handshake messages for verify data (DTLS)

Question

I am trying to create a DTLS client finished packet, where I need to generate verify data for handshake hash is need. And here I want to know how to generate handshake hash. As of now I am doing like this:

memcpy(hs_msg, client_hello_with_cookie+13, client_hello_with_cookie_len-13);
len = client_hello_with_cookie_len-13;
memcpy(hs_msg+len, server_hello+13, server_hello_len-13);
len += (server_hello_len-13);
memcpy(hs_msg+len, server_hello_done+13, server_hello_done_len-13);
len += (server_hello_done_len-13);
memcpy(hs_msg+len, client_key_exchange+13, client_key_exchange_len-13);

But the hash of handshakes seems to be wrong while comparing with openssl. (I am using sha1).

Need help to solve this...

Thank you very much.

Answer 1

First, I am assuming, per https://security.stackexchange.com/questions/29172/what-changed-between-tls-and-dtls, that the client handshake protocol in DTLS is not different from that in TLS over TCP. This seems a safe bet since the client/server encrypted handshake protocol in OpenVPN's UDP implementation is the same as in standard TLS over TCP.

I am not sure what you mean by "using sha1", since the client encrypted handshake message hash $V$ use a concatenation of MD5 and SHA1 hashes of the concatenated handshake (TLS opcode 0x22) messages. Details are in Davies' "Implementing SSL/TLS," pp. 345-350:

$$V= \text{PRF}(\text{master secret},\text{"client finished"},\text{MD5}(H)\|\text{SHA-1}(H),12)$$

$H$ is the concatenation of the handshake messages; "PRF" is TLS-PRF, a Python implementation of which follows:

def TLS_PRF(secret,label,seed,length):
    def P_HASH(secret,seed,length,h):
            i = 1
            A = [seed]
            out = ''
            while len(out) < length:
                    A.append(HMAC(A[i-1],secret,hasher=h))
                    out += HMAC(A[i]+seed,secret,hasher=h)
                    i += 1
            return out[:length]
    f=''
    md5 = P_HASH(secret[:len(secret)/2],label+seed,length,hashlib.md5)
    sha1= P_HASH(secret[len(secret)/2:],label+seed,length,hashlib.sha1)
    for m,s in zip(md5,sha1):
            f += "%c"%(ord(m)^ord(s))
    return f 

If you run the concatenated 0x22 messages through the complicated process noted above, you should get agreement with at least the hash in the client's plaintext EHM.

Calculation of the server EHM hash is slightly trickier. To $H$, the extant concatenated 0x22 messages, you will need to concatenate the decrypted 12 byte value $V$ contained in the client EHM. Do not forget, per the RFC, to prepend this 12 byte value with the message type (probably 0x14) and three byte length value (0xc) - so add 14 00 00 0c <12 bytes of V> to the $H$ used for the client EHM, run that through the PRF (using "server finished" for your label), and you should have your final hash value.

Calculation of DTLS Application Data MACs

This is in response to an additional question posed by OP in comments to this answer, concerning calculation of the Application Data (DTLS packet type 0x17) MAC values. According to RFC 4347:

The DTLS MAC is the same as that of TLS 1.1. However, rather than using TLS's implicit sequence number, the sequence number used to compute the MAC is the 64-bit value formed by concatenating the epoch and the sequence number in the order they appear on the wire. Note that the DTLS epoch + sequence number is the same length as the TLS sequence number.

Here is some Python code that implements this for a packet at the beginning of the DTLS session, with the starting SeqNum and EpochNum values. $D$ is the decrypted application data including the "decrypted" IV at the start of the message, and the MAC and padding at end of message (basing all of on this on how Wireshark dissects DTLS traffic).

    SeqNum="000000000001".decode("hex")    #passed over wire 
    EpochNum="0001".decode("hex")          #passed over wire
    PacketType="17".decode("hex")          #Application Data
    ProtocolVersion="feff".decode("hex")   #DTLS 1.0
    PacketLength="0400".decode("hex")      #Tricky - the length of AppData (as passed in clear over the wire) minus 0x20 bytes of SHA-1 MAC and padding
    D="0xf00ba..."                         #The decrypted blob

Then the HMAC will be:

HMAC(EpochNum+SeqNum+PacketType+ProtocolVersion+PacketLength+D[0x10:-0x20],hmacKey)

Don't forget the exclude the first 0x10 bytes (which will be the meaningless, "decrypted" IV) and the last 0x20 bytes (MAC+padding). Apart from not including/ordering EpochNum and SeqNum correctly, the only other big pitfall, as previously noted, is forgetting to adjust the PacketLength to reflect the fact that the MAC obviously doesn't include itself as part of the message (see Davies, p. 376 for a discussion of this point in traditional TLS context).