# Best method for checking integrity of 50 GiB files?

I'm making backups of compressed data (using ZPAQ), some compressed archives are large (50 GiB)

I was just going to use sha3sum -a 512, for example:

sha3sum -a 512 "Total_DOS_Collection-RELEASE#17-[1981-1995].zpaq"


Then storing the value and later periodically checking if the value of the backup is still the same

Then I started wondering if there would be a better/safer/more appropriate/effective method...

• Could you define what is better, safer, /more appropriate/effective method for you? Where do you store these files? SHA3 is not a recovery for errors. You may need RAID or multiple backups, that is out of CSE. – kelalaka Oct 24 '19 at 7:35
• sha3 will be slow as hell, use blake2 in a tree hash mode – Richie Frame Oct 24 '19 at 8:37
• I know SHA3 is not a recovery for errors, the idea is me being aware when the file has somehow become corrupted, by comparing the actual SHA3-512 to the SHA3-512 of the original file – Charles D. Ward Oct 24 '19 at 9:00
• The best method in my opinion, would be one that allows me to know if the file has become corrupt or not, just that. I think knowing the original SHA3-512 sum of a file would be enough for this, I'm looking for confirmation here. If there would be any kind of modification of the compressed archive (partial corruption, for example), the SHA3-512 sum would be different, right? And if no modification had been done to the compressed archived, the SHA3-512 would always be the same, right? – Charles D. Ward Oct 24 '19 at 9:01
• It would appear that bt2sum is a ready-made tool that computes the aforementioned blake2 tree hash using good defaults (namely using the number of CPU cores for parallel processing), if you feel standard b2sum is too slow, give this a try (and only turn to b2sum if the sha3 utility is too slow for you). – SEJPM Oct 25 '19 at 11:32