# How Can I decrypt this VNC Authentication handshake?

I'm trying to analyze a mobile application and I believe it is is using DES. Here is a sample of the authentication handshake.

{""id":"handshake","authentication":"vnc","authentication-challenge":"A489C9790FB7C3CE2E56868C788641FC"}

{"id":"handshake","authentication-passed":true}


For simplicity, the password is set to AAAAAAAA.

For authentication, it is identified as "VNC". This leads me to believe that they are using VNC authentication, as defined by RFC 6143.

https://tools.ietf.org/html/rfc6143#section-7.2.2

The documentation states:

The client encrypts the challenge with DES, using a password supplied
by the user as the key.  To form the key, the password is truncated
to eight characters, or padded with null bytes on the right.  The
client then sends the resulting 16-byte response


So if I understand correctly, If I were to encrypt "0xA489C9790FB7C3CE2E56868C788641FC" with the key of "AAAAAAAA" it should equal "0xD33BAD35B07EF500A82329749CEE9192". Is that correct?

Challenge contains:

$cat ~/Desktop/challange | xxd -p a489c9790fb7c3ce2e56868c788641fc  I try to run: $ openssl des -in ~/Desktop/challange  | xxd -p
Verifying - enter des-cbc encryption password:
53616c7465645f5f3f72b55c62bf2c9da805942f64f028b090cecc652dde


Which doesn't prepuce what I am hoping for, which is D33BAD35B07EF500A82329749CEE9192. What am I doing wrong?

EDIT:

I found this online.

http://www.vidarholen.net/contents/junk/vnc.html

I believe this might explain why I'm having issues.

challenge":"318E90D502A379A7B1583C35DB772138"}
{"id":"handshake","authentication-response":"1D214EF554123712A0DC9BCF003C27F4"}
{"id":"handshake","authentication-passed":true}


Ok, I got it. Just answering for anyone else who stumbles here. While not documented, RFB mirrors each bit. Using the key of all B's, I was able to get my result, as B is a palindrome.

• FYI you would still get different results with openssl des because OpenSSL enc defaults to salted password-based encryption (the key is derived from the password, it is not the pw, and is randomized) in CBC mode with padding (both wrong for your case). echo 318E90D502A379A7B1583C35DB772138 |xxd -r -p |openssl des-ecb -nopad -K 4242424242424242 |xxd -p (-K takes hex and ASCII 'B' is hex 42) gives desired result 1d214ef554123712a0dc9bcf003c27f4. – dave_thompson_085 Oct 31 '16 at 5:24

To be clear, as Dave Thompson is implying, the challenge response appears to be just a vanilla DES ECB with no nibble swapping as suggested by the original poster (I was also sidetracked by the information on the link). Note that the password BBBBBBBB would translate to  if nibble swapped, so is not actually any different than AAAAAAAA in that respect.

I was able to get the challenge response with the python code found at: https://github.com/RobinDavid/pydes

Just change the main to use the test data:

import binascii
key_str = "BBBBBBBB"
data= binascii.unhexlify('318E90D502A379A7B1583C35DB772138')
d = des()
r = d.encrypt(key_str,data)
print binascii.hexlify(r)