I am working on a proxy dll for DirectX 9, and eventually DirectX 11, which dumps textures uploaded via the cpu to files on the disk. Crucial to some of the features I wish to implement is the ability to identify which files correspond to which textures after relaunching the DirectX-powered program.

I didn't want to use memory addresses to identify the files due to the "random" nature of memory addresses between subsequent program launches.

I decided to identify the textures by hashing the raw data, essentially identifying them by their contents.

As such, my priority is not algorithm security, but rather the uniqueness of the hashes. I also need to be able to implement such an algorithm into my own code, or use a library with a lax license.

New contributor
Adolif4 is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
  • 3
    $\begingroup$ Since you say security issn't an issue, I assume you're asking about avoiding only accidental collisions, not intentional collisions? In that case, have you considered using a CRC with a proper polynomial (if in doubt though, CRC32c is often a good choice). However any hash function or checksum will be too slow if you need to run it at an extreme rate. Anyway, if it's not about security, then it's off-topic on this specific SE. $\endgroup$ – forest Jul 12 at 6:02
  • $\begingroup$ Oh, I apologize. I wasn't aware of that. My specialty is game programming, so I had to do quite a bit of Wikipedia reading just to write the question. Two different worlds. $\endgroup$ – Adolif4 Jul 12 at 6:10
  • 1
    $\begingroup$ Well, Game Development might help, especially if this is an XY problem. What's important though is knowing how low of a collision probability is required. Is $2^{-32}$ fine? Does it need to be $2^{-128}$? $\endgroup$ – forest Jul 12 at 6:18
  • $\begingroup$ Well, even though it was in a comment and not a "proper" answer, you already gave me a lead. Given that my biggest priority is minimizing the possibility of accidental duplicate hashes, I've come to believe that CRC64 might be the best option. Most of the sites where it is discussed seem to dislike it due to its supposed lack of standardization. That is, however, of no concern to me. I'm just looking to identify blobs of data based on their contents. $\endgroup$ – Adolif4 Jul 12 at 6:46
  • $\begingroup$ Please note that CRC64 is not automatically better than CRC32. There are many situations where the latter is better at detecting collisions than the former. This is a result of a CRC's so-called "burst detection". $\endgroup$ – forest Jul 12 at 6:54

Sounds like a perfect use case for xxHash

It is extremely fast, up to 20GB/s on modern systems, while providing a 64-bit output on 64-bit systems for superior collision resistance when compared to CRC32. It is not a cryptographic hash, but your use case should not need that.

If you need more collision resistance, they do have an experimental version that is also very fast.

Code is available in many languages and the license is quite permissive.

  • $\begingroup$ Is there a mathematical/algorithmic description of xxHash anywhere? $\endgroup$ – kodlu Jul 13 at 9:51
  • $\begingroup$ I think this is very risky advice. For most realistic collisions, a good CRC is far better than something as bad as xxHash. There's a reason CRCs are universally recommended for detecting accidental collisions. $\endgroup$ – forest 2 days ago

Your Answer

Adolif4 is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.