I am currently analysing a client-server application which apparently uses RC4. I've got the secret key and nonce and I was very surprised about the approach they encrypted the packets.

Since I am no cryptography expert I will explain what I did so that it works.

  1. I created a string with the full key (nonce + secret) and I append all the unencrypted bytes without clearing it at any time
  2. I RC4 the whole string (which obviously grows over time)
  3. I slice the resulting string so that all old bytes will be cut and only my "new bytes" are left

I can't imagine the server side maintains the whole byte history for every connected client, hence I wonder if it's some sort of RC4 algorithm / modification or if this is a custom implementation

  • $\begingroup$ I think that you've only got half of the protocol. Where does the key /nonce exchange happen? The answer to my question will probably go a long way to answering yours... $\endgroup$
    – Paul Uszak
    Commented Aug 19, 2017 at 0:24
  • $\begingroup$ Can you be clearer about which parts of this are key scheduling (the secret and the IV?) and which are encryption (applying the stream cipher)? The latter, applying the cipher, is independent of the contents of the data you're encrypting, so you don't need to remember what ata only how much. But better still, to continue using the same cipher all you need to do is remember the state of the permutation and i and j, you don't need to repeat an encryption to get back to that state. $\endgroup$
    – Rup
    Commented Aug 20, 2017 at 9:14

1 Answer 1


If I understood you correctly, the protocol you are analysing doesn't reinitialize the encryption before encrypting, so it seems like it is encrypting everything it send yet and sends only the part that "is new".

To answer your question, let's take a little excourse into stream ciphers (as RC4). These ciphers basicly generate a stream of random numbers which are then xored with the plaintext. To decrypt, you have to generate the key stream again and then xor it with the ciphertext.

$C = P \oplus K$

$P = C \oplus K$

($C$ is the ciphertext, $P$ is the plaintext, $K$ is the keystream and $\oplus$ is XOR)

One property of such ciphers is that to en- or decrypt one section is that you just need the related section of the keystream. So, in your protocol, following will likely happen:


  1. Calculates $K_{0}$
  2. Sends $C_0 = P_0 \oplus K_0$
  3. Calculates $K_1$
  4. Sends $C_1 = P_1 \oplus K_1$

Note that the client doesn't need $K_0$ nor $P_0$ to encrypt $P_1$.


  1. Calculates $K_0$
  2. Recieves $C_0$ and decrypts it using $P_0 = C_0 \oplus K_0$
  3. Calculates $K_1$
  4. Recieves $C_1$ and decrypts it using $P_1 = C_1 \oplus K_1$, the server doesn't need $K_0$ nor $P_0$ to decrypt $C_1$, so it doesn't have to do the operations you did while reverse engineering.
  • $\begingroup$ Yep that was the reason, I had to create a RC4 instance for the encrypting and decrypting stream. $\endgroup$
    – kentor
    Commented Aug 20, 2017 at 18:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.