First, I am assuming, per https://security.stackexchange.com/questions/29172/what-changed-between-tls-and-dtls, that the client handshake protocol in DTLS is not different from that in TLS over TCP. This seems a safe bet since the client/server encrypted handshake protocol in OpenVPN's UDP implementation is the same as in standard TLS over TCP.
I am not sure what you mean by "using sha1", since the client encrypted handshake message hash $V$ use a concatenation of MD5 and SHA1 hashes of the concatenated handshake (TLS opcode 0x22) messages. Details are in Davies' "Implementing SSL/TLS," pp. 345-350:
$$V= \text{PRF}(\text{master secret},\text{"client finished"},\text{MD5}(H)\|\text{SHA-1}(H),12)$$
$H$ is the concatenation of the handshake messages; "PRF" is TLS-PRF, a Python implementation of which follows:
def TLS_PRF(secret,label,seed,length):
def P_HASH(secret,seed,length,h):
i = 1
A = [seed]
out = ''
while len(out) < length:
A.append(HMAC(A[i-1],secret,hasher=h))
out += HMAC(A[i]+seed,secret,hasher=h)
i += 1
return out[:length]
f=''
md5 = P_HASH(secret[:len(secret)/2],label+seed,length,hashlib.md5)
sha1= P_HASH(secret[len(secret)/2:],label+seed,length,hashlib.sha1)
for m,s in zip(md5,sha1):
f += "%c"%(ord(m)^ord(s))
return f
If you run the concatenated 0x22 messages through the complicated process noted above, you should get agreement with at least the hash in the client's plaintext EHM.
Calculation of the server EHM hash is slightly trickier. To $H$, the extant concatenated 0x22 messages, you will need to concatenate the decrypted 12 byte value $V$ contained in the client EHM. Do not forget, per the RFC, to prepend this 12 byte value with the message type (probably 0x14) and three byte length value (0xc) - so add 14 00 00 0c <12 bytes of V> to the $H$ used for the client EHM, run that through the PRF (using "server finished" for your label), and you should have your final hash value.
Calculation of DTLS Application Data MACs
This is in response to an additional question posed by OP in comments to this answer, concerning calculation of the Application Data (DTLS packet type 0x17) MAC values. According to RFC 4347:
The DTLS MAC is the same as that of TLS 1.1. However, rather than
using TLS's implicit sequence number, the sequence number used to
compute the MAC is the 64-bit value formed by concatenating the epoch
and the sequence number in the order they appear on the wire. Note
that the DTLS epoch + sequence number is the same length as the TLS
sequence number.
Here is some Python code that implements this for a packet at the beginning of the DTLS session, with the starting SeqNum and EpochNum values. $D$ is the decrypted application data including the "decrypted" IV at the start of the message, and the MAC and padding at end of message (basing all of on this on how Wireshark dissects DTLS traffic).
SeqNum="000000000001".decode("hex") #passed over wire
EpochNum="0001".decode("hex") #passed over wire
PacketType="17".decode("hex") #Application Data
ProtocolVersion="feff".decode("hex") #DTLS 1.0
PacketLength="0400".decode("hex") #Tricky - the length of AppData (as passed in clear over the wire) minus 0x20 bytes of SHA-1 MAC and padding
D="0xf00ba..." #The decrypted blob
Then the HMAC will be:
HMAC(EpochNum+SeqNum+PacketType+ProtocolVersion+PacketLength+D[0x10:-0x20],hmacKey)
Don't forget the exclude the first 0x10 bytes (which will be the meaningless, "decrypted" IV) and the last 0x20 bytes (MAC+padding). Apart from not including/ordering EpochNum and SeqNum correctly, the only other big pitfall, as previously noted, is forgetting to adjust the PacketLength to reflect the fact that the MAC obviously doesn't include itself as part of the message (see Davies, p. 376 for a discussion of this point in traditional TLS context).
