Prove that you have the data when hash is public

Question

I am designing an online service where users can upload files (3d models) for processing. I want to speed up things when the same file is uploaded by many different users by allowing them to post a hash (SHA256) of the file before sending the file. If the hash exists they immediately get the processed file otherwise they upload a file.

I worry about this approach that a user can find SHA256 of a file somewhere he may be able to download the processed file without actually owning the file. E.g. SHA256 is somewhere public, but the file is private (e.g. a marketplace made SHA256 for paid 3d models public and someone paid for them and processed them using my service) ... or the metadata is hacked

The questions are:

1. Is this a genuine concern? Should I be worried about people getting SHA256 and abusing it to get other user's files?
2. How do I fight this? I was thinking of using HMAC: the server would send a key to the user and ask him to generate HMAC for the file and send it back?
3. Would it be safe to make HMAC key public so I don't store it for each upload request but the user can just generate HMAC based on this public key?

Answer 1

1. It's indeed a genuine concern.

2. How to fight this?

Well, once client had provided the SHA256 hash of the file (to quickly index the file on the server), your server then provide a random one-time key and ask the client to hash the file again with HMAC and the key. Since the key is one-time and random, adversaries have little chance at guessing the key and forging a MAC tag of a file that they only knows the hash of.

3. It wouldn't be safer to make HMAC key public, as it's equivalent to regular hashing with custom initialization vector.

== Some side notes ==

Browsers nowadays have WebCrypto APIs, which provide the functionalities of hashing and HMAC, but using them in a web page requires the secure HTTPS connection. Also, these APIs don't work on File/Blobs, they work on ArrayBuffers, which for large files could be memory intensive.

Answer 2

(1) Yes, it would be a concern. Once upon a time, Dropbox did this. When synchronizing the files, it checked the hashes before uploading. Thus, if the file was already in their servers, they just linked it and the transfer was instantaneous. Then people started using it for sharing content. By having the hash of e.g. a movie, they could "claim" to to have that file locally, and add it to their account. Then they would actually download the file from their Dropbox account. They ended up removing that optimization. Now the client must upload the file in full, then they will hash it on their servers and only then deduplicate.

(2) Your approach would work. Note they would have to provide the file hash first, so you can identify the file they claim to have. Then you send them a challenge (the salt of the hmac) and compare their answer while computing lt with the file you already have on your server.

On (3) I think you want to always have the same key for a given file. I feel this isn't a good idea (you don't know they do have the file, only that they know the ANIVGames' hmac of the file), although probably it won't really be a problem for you.

---

I think you are trying to prematurely optimize it. You are designing the service. I would make it work by uploading the file contents (ok, if you have the rendered output, you can reuse it). After everything works, I think it would be appropriate to start catching optimization oportunities like this.