MD5 hash functions to prove integrity of a download

Question

It doesn't take a lot of time on the crypto StackExchange to realise the sentiment around MD5 hash functions is poor.

My questions are:

• If it's such a bad idea to make use of MD5 hash functions why do big websites (such as the ubuntu.com) still make use of the hash function to verify the integrity of the file downloads?
• Is this simply the remains of a system that has not yet been updated?
• Or is this piggy-backing off of the fact that RSA encryption in the web-browser ensures (in most cases) that you are on the official ubuntu.com website?

Practically speaking, there are several ways a hash collision could be exploited. if the attacker was offering a file download and showed the hash to prove the file’s integrity, he could switch out the file download for a different file that had the same hash, and the person downloading it would be unable to know the difference. The file would appear valid as it has the same hash as the supposed real file. - https://learncryptography.com/hash-functions/hash-collision-attack

My current hypothesis is that although Oscar (attacker) can easily identify collisions and provide the victim some file that has the same hash output for this to be an issue bigger than merely an inconvenience on behalf of "the victim" by receiving the incorrect file, there would need to exist a malicious executable file that has the same hash output, which is less likely.

Answer 1

MD5 can be used like a checksum, ie. to detect accidental data corruption. It is not a good function to detect tampering.

Examples where comparing hashes might be useful even for non-cryptographic hashes:

• Dropped packets - Not a concern for files delivered over TCP, but if packets get dropped silently the download would be missing bytes. MD5 would almost certainly produce a different hash in that case.
• Client/server bugs - It is possible to specify that you want to download a smaller range of bytes instead of a whole file. This is why you can pause downloads or lose your connection and pick up the download without starting over. It's fairly uncommon now, but occasionally I find downloads get corrupted this way. Of course this is one specific bug, bugs in general also can cause problems that MD5 could detect.
• Typos and filenames - Accidentally type executable.v.2.6.112 instead of executable.v.2.6.121? If the file contents are different then the MD5 hashes will likely be different. It also helps when filenames on a download page are mistyped or a mirror for some reason uses different filenames.
• Mirror problems - Sometimes I download a file, notice that the file size is far too small. When I open the file in a hex editor I find out that I downloaded a 404 page or some ad-filled domain parking page.
• MITM proxies - Some "legitimate" services perform what's essentially a MITM attack. For example, workplace surveillance, censorship systems, AD injecting WIFI, anti-malware software. If an active MITM proxy does a sort of find-and-replace on network traffic, say replacing "bad words" with "*** *****" this could inadvertently corrupt your download.

The most likely reason, though, is probably legacy. People used to use MD5 to check download integrity (where "integrity" may be used in a cryptographic or non-cryptographic sense) so websites continue to include MD5 hashes because they have done it for as long as most people there can remember.

---

Now suppose you have an ideal cryptographic hash function with a sufficiently large output. Then you can distribute hashes on a trusted webserver and allow less-trustworthy websites to provide a mirror service.

To distribute malicious files under conditions where the attacker has no way of influencing what hashes appear on the trusted website (and that the user correctly compares hashes and not just the first few bytes) the only way the attacker can create a valid substitute file is to manage to do a pre-image attack.

I'm not aware of practical pre-image attacks on MD5. But I would still (usually) discourage its use because there are better alternatives.

---

For any type of file, but perhaps especially for open source stuff, it's also a good idea to use a collision resistant algorithm. (MD5 isn't one.) If a collision resistant hash function has enough output bits, then it is safe to say that there does not exist more than one ("real world") file that matches a published hash.

If there were multiple files corresponding to the same output, someone could say "You should publish trusted.iso. And also I am willing to be a mirror." They might do that if trusted.iso is benign and relevant to their website. Then later the attacker could start distributing malware.iso from his mirror (with the name trusted.iso obviously) where md5sum trusted.iso == md5sum malware.iso. Then he could exploit the users trust in the main website (and MD5 in this case) to deliver malware.

---

But I am not sure if I would list any file hashes for a website I maintained. If possible public key based authentication would be better. But that only works if users can legitimately trust your public key, and they do the authentication, and they have a correct authentication algorithm implementation. (Authenticated HTTPS is better than plain HTTP, and it's seamless, but still a risky alternative.)

Including hashes might create a false sense of security among the non-cryptographer user base. One reason I might use MD5 is because it's almost common knowledge that it's not secure. It's better in that respect than using "Secure Hash Algorithm" 1, 2, or 3. But I really don't know what strategy is least harmful.

Answer 2

It's a bad idea, creating a collision in MD5 is trivial and executeables are rich enough to allow arbitrary functionality of the collieing executeables. giving any two programs A and B it is trivial to create two programs A' and B' with the exact same respective behaviour as A and B but with the same MD5 hash. $MD5(A')=MD5(B')

However I do not know of a practical second preimage attack against MD5. So if an honest party publishes the MD5 you are probably safe, but are still better off using a modern hash.