Let's say I have fileA and fileB, both encrypted somehow, but in the same method with different keys.

If I xor them together, they will make a repeating pattern of 32 bytes length.

What does that say about the repeating pattern, the encryption pattern, and the likelihood of the unencrypted files to be identical?

Is there any way to extract the keys used to encrypt them?

enter image description here

Is there a way, other than to open up Ollydbg and try to reverse engineer the decryption procedure?

The program contacts a server, sends a certain unique code, and the server replies back with an md5 hash. Then the server sends back the encrypted file, which is easily decrypted by the program. I want to reverse engineer this program and encryption method.

This is not a homework question, this is a real situation that is tormenting me.

The decrypted plaintext is either Lua source or Lua VM bytecode, with a small possibility of being XML.

Edit: I now realize that the repeating sequence is probably two different MD5 hashes XOR'd with each other. Is there a method to analyze and try to extract the individual MD5 hashes?

  • $\begingroup$ Identifying the cipher is a big step in breaking it. By all means do fire up Olly and have a crack at it. But you will also find the key, and you won't have to do any codebreaking. You can always find one part of the XOR if you know the other, so you should probably XOR the result with the MD5 hashes that you've intercepted $\endgroup$
    – rath
    Apr 19, 2014 at 5:54
  • 4
    $\begingroup$ Re close votes: Analyzing / decrypting a block of data for a question that doesn't contain an explicit request to do so isn't a valid close reason. $\endgroup$
    – rath
    Apr 19, 2014 at 5:55

1 Answer 1


The two plaintexts are almost certainly identical. At the very least, any difference between them must be representable in 32 bytes, since that's as much information as the XOR of the encrypted files contains.

Assuming that the plaintexts are indeed identical, we can also see that changing the encryption key has a very simple and predictable effect on the ciphertext: every 32nd byte is affected the same way. Clearly, this cannot be a semantically secure encryption scheme.

Indeed, the most likely explanation of the phenomenon you observe is that:

  1. the two plaintext files are identical, and

  2. the encryption simply consists of taking a 32-byte key (or possibly the 32-byte hash of a key) and XORing it with each 32-byte block of the plaintext.

If true, this is a very weak cipher, and it should not be hard to break. A couple of methods to try would be:

  • If you can guess some part of the plaintext, you can immediately get the corresponding part of the key by XORing the guessed plaintext with the ciphertext. For example, if the plaintext is XML, there's a good chance it'll begin with something like:

    <?xml version="1.0" encoding="utf-8" standalone="no"?>

    With a bit of luck, this can reveal the entire key! Similarly, I believe Lua bytecode files should have a more or less fixed header that should also reveal at least part of the key.

  • Even if you can't directly guess the plaintext, you can make use of the fact that the key repeats, essentially making the cipher a many-time pad.

    One way to attack such a many-time pad is to use crib dragging. Basically, you guess an $n$-byte string that's likely to occur somewhere in the plaintext (such as, say, function in Lua code) and XOR it with each $n$-byte substring of the ciphertext. Each XOR will yield a candidate $n$-byte substring of the key, which you then XOR with the corresponding parts of the ciphertext. If the guess was wrong, you'll mostly get garbage, but if and when you guess right, the decrypted plaintext suddenly starts making sense.

    Another way to attack such a cipher is to split the ciphertext into 32-byte blocks, take one of the blocks and XOR all the other with it (which cancels out the key, leaving you with an XOR of the plaintext blocks), print out the XORed blocks as a hex+ASCII dump (as with xxd -c 32) and examine the columns. With the help of an ASCII table, you can infer a lot about the plaintext block you chose to XOR with the others (and thereby about the key).

Alternatively, if you're feeling lazy, you can just download xortool and feed one of the ciphertexts to it. With a bit of luck (and the right options), it'll spit out the key and the plaintext (or at least something close enough for you to finish the job by hand) for you.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.