Given an encrypted file, what are possible ways to find its encryption algorithm?

Heuristic ways are okay as well.

  • $\begingroup$ There is no general answer. $\endgroup$ May 6, 2015 at 19:35
  • $\begingroup$ I asked for possible ways. I know there won't be any general, work-all-the-times, exact answer. $\endgroup$
    – Azad
    May 6, 2015 at 19:38
  • $\begingroup$ Is it safe to assume you are only interested in modern ciphers? Also, do you want to place a limit on the size of the encrypted file? With unlimited data, there may be some things you can do. Also, by "encryption algorithm" do you mean "AES vs Serpent vs 3DES", that sort of comparison, or are you interested in modes of operation too? $\endgroup$
    – mikeazo
    May 6, 2015 at 19:42
  • $\begingroup$ You could look at the size of the file, you could look at characteristics at the beginning or end (e.g. markers indicating algorithm, length, signature method and encoding), but that would only cover a small subset of possible file formats and algorithms. For example, I could simply encrypt the file by XOR with a stream cipher with the key based on file name and password. You could check to see if decrypting it with a given cipher and password yields a file that makes sense, but using a different password-derived key algorithm or parameters effectively changes the cipher as well. $\endgroup$ May 6, 2015 at 19:46
  • 1
    $\begingroup$ One possible clue that would reduce the possibilities might be if you know the decrypted size of the file. Also, keep in mind that it may not even be a password, just a random 256-bit key. There's essentially no way to determine the difference between having the wrong key or the wrong algorithm if there are no structural clues. $\endgroup$ May 6, 2015 at 20:30

1 Answer 1


One of the important properties of a modern cryptographic algorithm is that it resist distinguishing attacks. Distinguishing attacks are any way to tell the encrypted data apart from random data. So if you can tell what the algorithm used was simply by looking at the ciphertext the algorithm is vulnerable to this attack.

Unless there is a header or such that tells you the algorithm and mode used, or a weak cipher was used, determining the algorithm used from the ciphertext is impossible.

  • 1
    $\begingroup$ impossible is a strong word, maybe something more along the lines of "not significantly easier than trying all possible keys with all possible algorithms" $\endgroup$ May 6, 2015 at 23:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.