2
$\begingroup$

I'm designing a data storage system that should be both able to deduplicate data that share the same encryption key, and be reasonably secure.

I know that this kind of encryption quality can be achieved by using AES in CTR mode with a synthetic IV derived from the HMAC of the file chunk - so that the contents are deterministic and still not leak information about the similarity of chunks.

While thinking about this problem, however, it seems to me that a good way to allow data deduplication is to encrypt the SHA-256 of the unencrypted file, and use that as a key to determine if the contents are the same.

This seems simpler for a variety of reasons - the first being that it's cheaper to check if a file chunk already exists on the server; The other is that while the encryption of the SHA-256 hash should still be deterministic, the actual chunk contents' encryption don't have to be. This also allows me to perform other non-deterministic filters - like compressing the file contents before sending it.

Are there any down sides or known attacks for this cryptographic approach?

==== EDIT: More info ====

A file will be divided into chunks; Each chunk will have its plain-text SHA-256 hash calculated (this may be lowered to SHA-1 though for storage space reasons); This hash will be encrypted with a deterministic encryption - probably AES-256 (but suggestions are very welcome). This encrypted hash will represent the file chunk into the abstract file storage system. At this point, the encrypted hash will be checked against the storage system, and checked if it exists already. If it doesn't, the chunk will be compressed, encrypted (I still haven't decided what encryption will be used for this), and sent to the storage. The file storage system will be "dumb" - meaning that it won't make any check.

The chunk is not of fixed size - instead, it should implement a bup-like hashplitting, so deduplication can work better. I'm studying the possibility of having bigger chunks, but right now in bup's implementation, each chunk has about 8k of data.

Improve this question
$\endgroup$
6
  • $\begingroup$ Where does your hash reside in its encrypted state and will the same hash always encrypt to the same result? Perhaps a bit more about the plan in its current state (to include chunk size and boundaries) would help us provide an answer to downsides and inefficiencies (storage systems need to be fast as a rule). $\endgroup$
    – zedman9991
    Jan 7, 2015 at 14:41
  • 3
    $\begingroup$ Should probably be migrated to Cryptography SE. $\endgroup$
    – RoraΖ
    Jan 7, 2015 at 15:03
  • $\begingroup$ @zedman9991, thanks! I've added more information on the question now. Let me know if you need any clarifications! And thanks! $\endgroup$
    – Waneck
    Jan 7, 2015 at 15:27
  • $\begingroup$ @raz, thanks. I wasn't aware of the Cryptography SE. I don't think I can migrate myself, but if it's more on topic there, I guess it should indeed be migrated $\endgroup$
    – Waneck
    Jan 7, 2015 at 15:29
  • 1
    $\begingroup$ Note that downgrading to SHA-1 is only useful to save some CPU cycles. If you just want less bits, then simply use SHA-512 (faster than SHA-256 on 64 bit systems) and cut off the rightmost bits you don't want (or use SHA-512/224, for instance). $\endgroup$
    – Maarten Bodewes ♦
    Jan 7, 2015 at 22:00

1 Answer 1

Reset to default
1
$\begingroup$

Information leakage in systems that do data deduplication typically involves timing hints that the data is already stored and related leverage of that information. See the comments in https://security.stackexchange.com/questions/7142/online-backup-how-could-encryption-and-de-duplication-be-compatible/7146#7146 for insights there. Suggest you consider your attacker's profile and see of any of those issues apply as a confidentiality threat.

Efficient deduplication needs to maximize the capture of identical data and thus is typically done at the file level or some subset that will always produce the same chunks in the same file. If your chunk boundaries plan addresses that you are good otherwise consider it.

Finally, encrypting a hash with a method that always produces the same output would appear to add very little protection if any. The hash is by definition not reversible so the only potential value is in protecting against someone who can recreate the hash and wants to know if they are correct about the guessed plain text or binary. That attack assumes the attacker can see the resulting hash. If that is the case then seeing the predictably encrypted result adds very little to the attackers task as he only needs to see that the data is not moved to storage but rather already exists in the system (see first point about information leakage).

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for the overview. It helped to decide on a good system. We've ended up not encrypting the SHA's, but rather encrypting the SHA's in the metadata descriptor. $\endgroup$
    – Waneck
    Jan 11, 2015 at 18:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.