I am trying to reverse engineer a program that encrypts a file with a password.

So far i figured out that the start of the file is always the string "DecryptOK" encrypted with the password of course.

With a password of "one" an "empty" file turns into


which i assume would be

ED5B DE97 BB90 DE13 7794 8A31 B0EE CB4B 4D8B

aka "DecryptOK".

So my question is how do i continue from here? How do i figure out what encryption method is being used?

Here are some texts i encrypted with the password "one":

  • 123 to 3366 7FAD CAF5
  • abc to 3336 7FFD CAA5
  • cookies to 3334 7FF0 CAA9 33F7 D2D2 60AB 86EB
  • $\begingroup$ Requests to decrypt ciphertext are considered-offtopic on this site. Therefore this question will be closed most likely, or possibly even deleted: You are trying to break an actual program, and from the question statement my guess is that this is an illegal activity. $\endgroup$ – tylo Oct 20 '16 at 12:50
  • $\begingroup$ I voted to close this as too broad instead of as "requests to analyze ciphertext..." because in my opinion, the OP is not asking us to analyze any ciphertext. See an earlier question I posed on meta. That said, for the question "how do I continue" and "how does one figure out what encryption method is being used" given black box access to encryption, that is too broad of a question for here. There is no correct answer to this as it depends on so many specifics. (1/2) $\endgroup$ – mikeazo Oct 20 '16 at 14:30
  • $\begingroup$ In other words, there really isn't a formal process we can describe to answer that sort of a question. A specialist could start with some simple questions maybe, and start to drill down deeper and go in very different directions depending on what they find. Furthermore, this type of analysis is often not necessary in practice as we would have access to at least the binary which we could reverse engineer. The latter often being a much more fruitful plan of attack. (2/2) $\endgroup$ – mikeazo Oct 20 '16 at 14:32

So my question is how do I continue from here? How do I figure out what encryption method is being used?

It sounds like you, in essence, have black box access to something that allows you to give it a key and a file, and it returns an encryption of that file (after adding some file format around it) using the supplied key.

Your goal is to figure out how the encryption works. There may be legal issues you want to consider before actually doing this, but legal issues aside, you have many options.

Your best option would be to reverse engineer the code that is doing the encryption. Is this software running on your computer or some sort of hardware device? In either case, you should be able to get the code and reverse engineer it by putting it in a disassembler and doing static analysis, or putting it in a debugger and doing dynamic analysis.

Let's pretend, however, that you do not have access in any way to the code. Maybe this black box is running on a remote server that you do not have access to. Your goal then is to figure out what cipher are they using (e.g., AES, DES, 3DES, RC4, etc) and what mode of encryption is being used.

You can start to do this by asking a few questions and using your black box access to start answering the questions.

  1. Is the encryption deterministic? Or in other words, given the same plaintext (or file) and key, does it always return the same result?

  2. If I flip a single bit in the plaintext, how does the resulting ciphertext change?

  3. What happens if you feed a ciphertext in as the plaintext, do you get the original plaintext back? In other words, is the encryption mode the inverse of itself?

  4. By flipping bits at different places, can you figure out the block size? Depending on the mode used, where you flip bits can change what blocks actually change?

  5. Something else to investigate is why does the ciphertext appear to always be double the size of the plaintext? When you pass in 123, which as ASCII would have 3 bytes, why do you get 6 bytes back? Is there some sort of encoding going on?

This is how I would suggest continuing from where you are.


Not the answer you're looking for? Browse other questions tagged or ask your own question.