0
$\begingroup$

I'm looking for the most efficient in memory and time way to verify integrity of large files which have to be encrypted as well.

  • Using SHA256 and other crypto hashes over the whole encrypted file?
  • Split it into blocks and encrypt+hash each one separately?
  • Use authenticated encryption? Again, block-wise or as a whole? GCM or are there any better options today?
  • Something else?

Thanks!

$\endgroup$
  • 1
    $\begingroup$ I like Merkle trees. $\endgroup$ – CodesInChaos Feb 13 '18 at 14:56
0
$\begingroup$

I am going to have to say splitting it into blocks. Although your definition of large may differ from mine, GCM is unsuitable for large files, and hashing the entire file can be problematic, especially if the file is spread across storage devices, and parts of it are not currently available for reading in the sequence you want.

Splitting it into blocks, say 4-16MB at a time, will incur very little overhead for storage of a hash, allow you to encrypt several blocks at a time in whatever order is currently available, and allow you to verify integrity of selected data out of sequence.

Each large file would be given an index file, which contains the IVs and integrity hashes for each block, which can then be encrypted and authenticated independently of the blocks. If you have verified the index data, you can perform random reads of the blocks by validating their hash and decrypting them.

As long as you are using secure methods for hashing, authenticating, and encrypting, (say SHA-512, HMAC-SHA-512, AES-CTR) the full system should be secure, and be very simple to implement, in addition to being extremely fast and being able to handle extremely large files. At least 36 index bytes would be required per block depending on implementation, so a 1TB file with 16MB blocks would require a 2.25MB index file, 100TB 225MB, and so on, and the ability to scale up to 64PB per large file, using a 144GB index, which could also be broken into blocks using the same method if necessary.

You would be looking at around a 60ms latency or less to access any 16MB block of the large file due to encryption/hashing overhead, if the index file is already authenticated. Smaller blocks would give lower latency but larger index files. Spinning storage latencies are larger and there is overlap between read and decrypt, so the performance hit on a modern cpu would be much smaller, probably well under 10ms per block, as decryption can be performed at the same time as hashing with the same read, in 1KB increments, and the decryption speed is not the limiting factor. When using faster storage such as NVME drives, a hash tree mode will improve performance if you have CPU cores to spare.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.