I am designing an online service where users can upload files (3d models) for processing. I want to speed up things when the same file is uploaded by many different users by allowing them to post a hash (SHA256) of the file before sending the file. If the hash exists they immediately get the processed file otherwise they upload a file.

I worry about this approach that a user can find SHA256 of a file somewhere he may be able to download the processed file without actually owning the file. E.g. SHA256 is somewhere public, but the file is private (e.g. a marketplace made SHA256 for paid 3d models public and someone paid for them and processed them using my service) ... or the metadata is hacked

The questions are:

  1. Is this a genuine concern? Should I be worried about people getting SHA256 and abusing it to get other user's files?
  2. How do I fight this? I was thinking of using HMAC: the server would send a key to the user and ask him to generate HMAC for the file and send it back?
  3. Would it be safe to make HMAC key public so I don't store it for each upload request but the user can just generate HMAC based on this public key?
  • $\begingroup$ Considering that the subject is an image or other file, you should note that even the smallest change in the file will cause a complete change in the output (hash value), and if, for example, I am one of your users or customers and I realize That you designed this mechanism to check for integrity, I will create an easy solution and changing a pixel in that file and re-upload it again and it could work. $\endgroup$ – R1w Nov 13 '20 at 1:40
  • $\begingroup$ The SHA-255 is enough for you to check the data process earlier, but your real concern is not that as I've shown in my answer and that addresses your real problem! Except for the law, noone cannot prevent the data and result share. $\endgroup$ – kelalaka Nov 13 '20 at 13:27
  • $\begingroup$ Nothing can solve the real issue except the LAW! $\endgroup$ – kelalaka Nov 13 '20 at 14:18
  1. It's indeed a genuine concern.

  2. How to fight this?

Well, once client had provided the SHA256 hash of the file (to quickly index the file on the server), your server then provide a random one-time key and ask the client to hash the file again with HMAC and the key. Since the key is one-time and random, adversaries have little chance at guessing the key and forging a MAC tag of a file that they only knows the hash of.

  1. It wouldn't be safer to make HMAC key public, as it's equivalent to regular hashing with custom initialization vector.

== Some side notes ==

Browsers nowadays have WebCrypto APIs, which provide the functionalities of hashing and HMAC, but using them in a web page requires the secure HTTPS connection. Also, these APIs don't work on File/Blobs, they work on ArrayBuffers, which for large files could be memory intensive.

  • $\begingroup$ This still doesn't protect file and result sharing. The OP's real problem can only be solved with Law! $\endgroup$ – kelalaka Nov 13 '20 at 13:28
  • $\begingroup$ It doesn't protect users from uploading the whole processed files somewhere else, but it makes it impossible for users to obtain the file if they know the hash of the original. $\endgroup$ – ANIVGames Nov 13 '20 at 13:55
  • 2
    $\begingroup$ @ANIVGames the users who want to pay less decides to open a shared-site where they put the file hash and the result of your process. So, when a person decided that they need to process their file with your software, they will first check the site. If there they will pay less, if not they will use your service then they will share their file, too. This is a win-win for them and lost-lost for you. The only protection against this is the LAW! $\endgroup$ – kelalaka Nov 13 '20 at 17:14

(1) Yes, it would be a concern. Once upon a time, Dropbox did this. When synchronizing the files, it checked the hashes before uploading. Thus, if the file was already in their servers, they just linked it and the transfer was instantaneous. Then people started using it for sharing content. By having the hash of e.g. a movie, they could "claim" to to have that file locally, and add it to their account. Then they would actually download the file from their Dropbox account. They ended up removing that optimization. Now the client must upload the file in full, then they will hash it on their servers and only then deduplicate.

(2) Your approach would work. Note they would have to provide the file hash first, so you can identify the file they claim to have. Then you send them a challenge (the salt of the hmac) and compare their answer while computing lt with the file you already have on your server.

On (3) I think you want to always have the same key for a given file. I feel this isn't a good idea (you don't know they do have the file, only that they know the ANIVGames' hmac of the file), although probably it won't really be a problem for you.

I think you are trying to prematurely optimize it. You are designing the service. I would make it work by uploading the file contents (ok, if you have the rendered output, you can reuse it). After everything works, I think it would be appropriate to start catching optimization oportunities like this.

  • $\begingroup$ This is a great answer as well, showing the solution from Dropbox. $\endgroup$ – ANIVGames Nov 13 '20 at 13:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.