Imagine the following situation:
- there are many "big" files (100Gb+) lying on 2 different remote servers
- you have no direct access to these files
- you can only read a small amount of each file (e.g. 10Mb) as well as it's general information (e.g. file size)
- files can have different names and paths but same content/size
- security is no concern - nobody will try to create fake data to force collisions
- file contents are solely audio and video files in all kinds of different formats
What's the best way to tell if there are duplicate files between these 2 servers (with a certain confidence) with the data that's available?
Goals:
- find file duplicates
- the resulting hash has to be the same for the same filesize + contents everytime
- create a database of hashes (?) to compare newly added files with already existing for duplication
My naive approach looks like this right now:
- feed file size (as byte array) into sha256
- read chunk at the beginning of file and feed into sha256
- read chunk at middle of file (file size/2) and feed into sha256
- read chunk at end of file and feed into sha256
- use resulting hash for comparisons
The goal is not to 100% identify/verify those files but being able to tell with the highest possible confidence (provided the restrictions) that file1 on server1 is pretty likely file2 on server2 while providing good collision resistance.