Determine Encryption Used by Examining File Headers?

Question

I have an encryption problem that I hope you might be able to help me with.

I have a bunch of encrypted files. Some of them were encrypted with mdecrypt.rijndael-256 in PHP and some were encrypted with mdecrypt.tripledes. A database used to exist that mapped the filename of each encrypted file to the appropriate encryption algorithm used for that file. That database has long been wiped from the server and now I only have the encrypted files that are either AES or 3DES. So my question is, is it possible to determine which algorithm was use to encrypt each file by examining a header or something? An added bonus would be if you could provide a example of how to determine the algorithm using a php script (although that's secondary and I can probably figure that out if I know what to look for).

Thanks.

Answer 1

That depends. A file cannot be directly encrypted by a block cipher. You need at least a mode of operation for that block cipher and an IV. I guess you should first determine the correct encryption format that was used.

If your file is directly encrypted you could try and decrypt using one of the two methods. If the result looks like a file then the right mode must have been chosen. You can e.g. test this by using the file command on Posix systems after decrypting the first kilo byte of the file (into a separate, partial file).

Alternatively you could perform a simple randomness test on the result. If that fails, then it is likely a correct file and the correct algorithm. This may however lead to more false positives than the method used above. You may want to test which of the partially decrypted files is least random and chose that algorithm.