Take the 2-minute tour ×
Cryptography Stack Exchange is a question and answer site for software developers, mathematicians and others interested in cryptography. It's 100% free, no registration required.

Why there is a lot of duplicate file finder application are using MD5 Algorithm? What is the strength of MD5 in terms of searching duplicate files in hard disk or flash driver or any other storage device?

share|improve this question
add comment

2 Answers

up vote 8 down vote accepted

MD5 is ok here as usual cryptographic attacks do not apply in this scenario. The probability of accidental MD5 collision is much less than usual probability for soft error. For details read more.


MD5 is currently considered too weak to work as a cryptographic hash. However, for all traditional (i.e. non-cryptographic) hash uses MD5 is often perfectly fine.

MD5 has one important benefit above SHA1, SHA2, and upcoming SHA3: many platforms allow faster implementation of MD5 than other hashes. The performance in this use case varies highly according to used hash function.

The practical attacks against MD5 are mostly against collision resistance. If these are not an issue, MD5 can be fairly ok. (See earlier question Is a second pre image attack on MD5 feasible?.) For instance, where MD5 is considered fine by many is HMAC-MD5.

To get back to this question. It can be assumed that find duplicate files is not something where collision resistance is such of an issue (you likely do not have $2^{64}$ files), so chances of collision are much smaller than in many cryptographic uses where it is assumed adversary tries to find collisions.

Thus, for usual hash uses, the space of MD5 (128 bits) is large enough. The probability of collision in usual situations is small enough, it is much larger probability to for example, detect false duplicates due to random hardware failure (like soft error).

For more accurate mathematics, see earlier question What is the probability of md5 collision if I pass in 2^32 sets of string?. This is applicable in scenario where user has less than four billion files, and you may use the formula for other values. One million files gives $2^{-89}$, which I would consider very small.


Even more efficient for duplicate file finding than MD5 is to consider all file sizes. Files which are not the same size are not duplicates. Only files that have size which is not unique need hashing. If file duplicate program using hashing wants to be safe from MD5 hash collision, the program may want to perform byte-per-byte comparison to ensure files are unique.

share|improve this answer
add comment

MD5 is less CPU intensive than other cryptographic hash functions. For example, the SHA1 algorithm is more complex than MD5 meaning it is less efficient to calculate. This may be the reason MD5 is used.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.