This apparently was a bug with WhatsApp a while ago. I am endeavoring to recreate it and understand it.
The bug had to do with the fact that the same key was used to encrypt plaintext on both the client and server sides. However,
(A ^ X) ^ (B ^ X) = A ^ B ; ^ is the bitwise XOR function. In other words, if if there are two messages encrypted with the same RC4 key, the key stream cancels out.
To illustrate this with an example: we have a simulator which creates a bunch of messages for both the server and client side. For the sake of simplicity, assume that all the messages are 128 bytes long (i.e. they are always padded to ensure they are that length) so that we do not have to guess where messages start and end.
An example message from the client side is:
LOGIN JOHN foo //padding
An example message from the server side is:
INVALID PASSWORD //padding
I don't know either of these messages.
Say that with a bit of traffic sniffing, I've come into the possession of two files,
Reading the first byte from
ServerSideEncrypted.dat I discover that
Byte1SS_encrypt = 01111100 and reading the first byte from
ClientSideEncrypted.dat I discover that
Byte1CS_encrypt = 01111001. Now,
01111100 ^ 01111001 = 00000101.
To ensure that this checks out, assuming that I did know the first letters of each of the messages from the client and server sides, I would have
Byte1CS_plain (the letter L) = 76 (in ascii) = 01001100 and
Byte1SS_plain (the letter I) = 73 (in ascii) = 01001001.
01001100 ^ 01001001 = 00000101. So indeed, the XOR of the two encrypted bytes is equivalent to the XOR of the two plaintext bytes.
At this point is where I become stuck. I know the following things:
Byte1CS_plain ^ Byte1SS_plain = Byte1CS_encrypt ^ Byte1SS_encrypt = 00000101 Byte1CS_encrypt = 01111001 Byte1SS_encrypt = 01111100
However, this still leaves me with 2 unknowns. How do I proceed with this information to decipher the messages? How do I figure out that
Byte1CS_plain = 01001100 and
Byte1SS_plain = 01001001?