Checking a file for integrity and authentication

Question

A checksum is a small-sized datum derived from a block of digital data for the purpose of detecting errors which may have been introduced during its transmission or storage. It is usually applied to an installation file after it is received from the download server. By themselves, checksums are often used to verify data integrity but are not relied upon to verify data authenticity.
[wikipedia]

1. How would you verify data authenticity then? Would transferring a file through the https protocol and then comparing it's hash confirm that it's genuine, or is there more to it than that?

2. And how would you hash a big 40+gb zip file, would you have to hash the entire contents of the file and, each file within the zip file?

I couldn't find any good resources on how to hash verify big files.

Answer 1

This relate to the difference between authenticity and integrity.

• Authenticity:
  you are sure of the identities involved:
  Where does the data come from?

• Integrity:
  you are sure that the content has not been modified:
  Was there an error during the transmission?

The second is usually provide through a Checksum (such as any hash function) e.g. $$SHA-3(\ M\ )$$

while Authenticity requires a MAC (Message Authentication Code), often in a form of a HMAC or KMAC e.g. $$SHA-3(\ K\ ||\ M\ )$$ or a signature e.g. $$ Sign_{K_{priv}}(\ SHA-3(\ M\ )\ ) $$

Remark:$$\textbf{Authenticity} \implies \textbf{Integrity}$$

---

As for your second question:

You have to hash the whole file, hashing sub contents will increase the workload but will provide you integrity over small part and allow you to detect which part has been corrupted.

Note:
To hash big files you might want to have a look this article: Is SHA-3 slow ?

Further readings:

• What are the differences between a digital signature, a MAC and a hash?
• What is the advantage of digital signatures over message authentication codes?
• How is digital signature different from a message authentication code (MAC)?

Answer 2

For the first question, if you connect to a website using HTTPS and the website is authenticated using valid certificate, then this guarantees that you are connecting to the genuine website and data coming from this website is from the genuine source. HTTPS should also guarantee that the transferred data are not being modified during transit. But using Checksum for a file will guarantee that the data provided by this authenticated source has not been modified (even by this authenticated source). e.g. if a third party server is uploading a Linux distribution to you, and you have that distribution checksum obtained from Linux website (the original source). Once you download the distribution from this third party, you compute the checksum, then compare it to the one from Linux. If you get the expected checksum, this guarantee you that the third party did not corrupt or modify the Linux distribution that it is providing you.

For the second question, there are wide variety of tools that compute the checksum over large files:

• md5sum (depreciated)
• sha1sum (depreciated)
• sha256sum
• sha512sum