Can multi-encrypted file be decrypted with one password only?

Question

Let's say I have a single file I want to encrypt with AES and I encrypt it twice (first encrypt the unencrypted file, then encrypt the encrypted file again). The first time I use password 'ABC', the second time I use 'DEF'. So in theory the attacker would have to decrypt it first by providing 'ABC' as password which would result in still encrypted garbage data - so they would provide 'DEF' and then they succeed. I wonder, though, if they may decrypt my file with only one password - perhaps there exists a password that is neither 'ABC' nor 'DEF' that successfully decrypts the file and they accidentally succeed with this password when bruteforcing.

In other words:

ABC -> DEF -> plain file

XYZ (exists or not?) -> plain file

Does the existence of 'XYZ' depends on the algorithm(s) used and will it always/never/sometimes exist?

Answer 1

There are two areas of concern here.

First,

Your password isn't the actual crypto key. The crypto key is probably derived from the password using a key derivation function. With a bad function, there is a risk that two different passwords could result in the same key. However, since the key is typically longer than the password, this risk is close to nil if the function is designed properly, because the output has more entropy than the input. KDFs are the opposite of hashes in this respect.

Second,

It could be possible for a single key to have the same effect as two keys applied in series. For example, if using a simple XOR algorithm, XOR'ing the contents with 64 and then with 128 would be the same as XORing it with 192.

Fortunately, AES does not use a simple XORing algorithm. It does use XOR, but only as one of several steps. Other steps swap bytes around or shift them up and down in a matrix. Thus there is no risk of a single key having the same effect as two keys applied in series due to the way the algorithm is designed.

I suppose there may be a risk that by sheer coincidence a single crypto key could decrypt the whole mess in one go-- I don't know how you'd find it. There would be only one such key, and it would take millions of years to look for it.

Answer 2

Theoretically speaking, there is a possibility that a specific hashing algorithm can hash two separate passwords that both result to the same hash, however the odds of such an event making much of a difference in a brute force attempt is minimal. In addition, if there are two separate file containers (one inside the other), a brute force attempt would stop after decryption the outside container. My recommendation for encryption would be AES-256 encryption with SHA-512 hashing as those are currently accepted as industry leading algorithms. Should you care to, veracrypt will give you an option nest encryptions such as Serpent-Twofish-AES which will in turn make decryption take more time.

Answer 3

Possible collisions aside, Mathematically there may be a single solution for any particular round because of how each round is generated(sub-lookup, shift, mix, add round key) there would be no relationship of one round key to any previous round only forward once a single round key was stumbled upon. Statistically determining the key for each round would difficult and time-consuming approaching impossibility for the total number of rounds.