Not sure if this falls into crypto from contextual point of view but it is about hashing algorithms. I have two directories -- assets/ and cache/. Anytime there is a file added, deleted, or changed in the assets/ directory, a corresponding, application specific file will be generated in the cache/ directory. On top of that, an additional "cache file" gets created that stores the following information:

assetFile: my-file.png # relative to assets directory
cacheFile: my-file.ktx2 # relative to cache directory
assetHash: 709a0aef5d1ecda90fb3f3542aa71bef08b9fab8 # hash from contents of asset file
cacheHash: 0511420356589c5669c83daeff059d68078aef80 # hash from contents of cache file

These files are generally huge. Some of these files can be 15-20mb in size.

The general purpose is not about security or privacy but it is more about checking if the source data is changed. So, in the example above, if my-file.png is changed, the application will check the file's contents against the hash stored in the "cache file" and if the hashes do not match, it will recreate my-file.ktx2 and update the "cache file" with a new hash. The assets are generally added by the user themselves; so, if they try to willfully tamper with this system, they will be breaking their own workflow.

What kind of hashing algorithm can I use here that is fast enough to create the hash but also reliable enough in terms of not having false negatives (i.e collisions)?

I am currently using Sha256 hash and it is quiet slow, especially on large files.

  • 4
    $\begingroup$ Are you sure you're bottlenecked by hashing the file, and not by reading it from disk? $\endgroup$ Nov 5, 2022 at 10:13
  • 2
    $\begingroup$ You don't have to worry about collisions when using any modern hash function with a 256-bit output. For large files, accelerated BLAKE3 is probably the fastest you can get. However, there's an issue with some BLAKE3 implementations that causes bugs with large large files (e.g. 2 GiB). Also, SHA-2 with acceleration is fast too. $\endgroup$ Nov 5, 2022 at 10:23
  • 1
    $\begingroup$ "user themselves; ... tamper with this system". Upon reading this part, randomized hashing using a static secret key came into my mind (think HMAC). $\endgroup$
    – DannyNiu
    Nov 5, 2022 at 12:02
  • 2
    $\begingroup$ @Gasim Yes, there are different instruction sets that provide hardware acceleration, improving performance. In the grand scheme of things, these are not large file sizes. You should investigate the performance of how you're reading from disk and the SHA-2 implementation you're using. I have a program for hashing with various algorithms, and hashing a 35 MiB file with SHA-256 is practically instant. $\endgroup$ Nov 5, 2022 at 19:11
  • 1
    $\begingroup$ As long as you stream or memory map it. It's amazing how many file hash & encryption routines on e.g. Stack Overflow simply read things in memory first (and if you are unlucky, perform some interesting and unnecessary encoding actions on top of that, it seems like a minimum of 3 in memory copies is the minimum for some reason or other). $\endgroup$
    – Maarten Bodewes
    Nov 6, 2022 at 11:11


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.