Incremental RC4 algorithm?

Question

I am currently analysing a client-server application which apparently uses RC4. I've got the secret key and nonce and I was very surprised about the approach they encrypted the packets.

Since I am no cryptography expert I will explain what I did so that it works.

1. I created a string with the full key (nonce + secret) and I append all the unencrypted bytes without clearing it at any time
2. I RC4 the whole string (which obviously grows over time)
3. I slice the resulting string so that all old bytes will be cut and only my "new bytes" are left

I can't imagine the server side maintains the whole byte history for every connected client, hence I wonder if it's some sort of RC4 algorithm / modification or if this is a custom implementation

Answer 1

If I understood you correctly, the protocol you are analysing doesn't reinitialize the encryption before encrypting, so it seems like it is encrypting everything it send yet and sends only the part that "is new".

To answer your question, let's take a little excourse into stream ciphers (as RC4). These ciphers basicly generate a stream of random numbers which are then xored with the plaintext. To decrypt, you have to generate the key stream again and then xor it with the ciphertext.

$C = P \oplus K$

$P = C \oplus K$

($C$ is the ciphertext, $P$ is the plaintext, $K$ is the keystream and $\oplus$ is XOR)

One property of such ciphers is that to en- or decrypt one section is that you just need the related section of the keystream. So, in your protocol, following will likely happen:

Client-Side:

1. Calculates $K_{0}$
2. Sends $C_0 = P_0 \oplus K_0$
3. Calculates $K_1$
4. Sends $C_1 = P_1 \oplus K_1$

Note that the client doesn't need $K_0$ nor $P_0$ to encrypt $P_1$.

Server-Side:

1. Calculates $K_0$
2. Recieves $C_0$ and decrypts it using $P_0 = C_0 \oplus K_0$
3. Calculates $K_1$
4. Recieves $C_1$ and decrypts it using $P_1 = C_1 \oplus K_1$, the server doesn't need $K_0$ nor $P_0$ to decrypt $C_1$, so it doesn't have to do the operations you did while reverse engineering.