I'm making backups of compressed data (using ZPAQ), some compressed archives are large (50 GiB)

I was just going to use sha3sum -a 512, for example:

sha3sum -a 512 "Total_DOS_Collection-RELEASE#17-[1981-1995].zpaq"

Then storing the value and later periodically checking if the value of the backup is still the same

Then I started wondering if there would be a better/safer/more appropriate/effective method...

  • 1
    $\begingroup$ Could you define what is better, safer, /more appropriate/effective method for you? Where do you store these files? SHA3 is not a recovery for errors. You may need RAID or multiple backups, that is out of CSE. $\endgroup$ – kelalaka Oct 24 '19 at 7:35
  • 2
    $\begingroup$ sha3 will be slow as hell, use blake2 in a tree hash mode $\endgroup$ – Richie Frame Oct 24 '19 at 8:37
  • $\begingroup$ I know SHA3 is not a recovery for errors, the idea is me being aware when the file has somehow become corrupted, by comparing the actual SHA3-512 to the SHA3-512 of the original file $\endgroup$ – Charles D. Ward Oct 24 '19 at 9:00
  • $\begingroup$ The best method in my opinion, would be one that allows me to know if the file has become corrupt or not, just that. I think knowing the original SHA3-512 sum of a file would be enough for this, I'm looking for confirmation here. If there would be any kind of modification of the compressed archive (partial corruption, for example), the SHA3-512 sum would be different, right? And if no modification had been done to the compressed archived, the SHA3-512 would always be the same, right? $\endgroup$ – Charles D. Ward Oct 24 '19 at 9:01
  • 1
    $\begingroup$ It would appear that bt2sum is a ready-made tool that computes the aforementioned blake2 tree hash using good defaults (namely using the number of CPU cores for parallel processing), if you feel standard b2sum is too slow, give this a try (and only turn to b2sum if the sha3 utility is too slow for you). $\endgroup$ – SEJPM Oct 25 '19 at 11:32

Tree hashing would be a natural way to do it.

However, the data still needs to be read. And in practice the hash computation time is negligible compared to I/O, even with a fast SSD.

So, just use a standard hash function. SHA-3 is slow in software, so you'd better opt for BLAKE2.

  • $\begingroup$ Dunno, SSD's have become pretty fast of late. I've got an Intel 660p which runs at 1.8 GB/s at full speed. Sure, hashes can be fast, but at those speeds you'll have to make sure that you've got the right implementation at the very least. $\endgroup$ – Maarten Bodewes Oct 25 '19 at 10:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.