Skip to main content
added 1453 characters in body
Source Link
Schwern
  • 461
  • 4
  • 9

You say you want to decompress the data coming from A so B can do incremental backups and recovery. Were A's data not encrypted this would make perfect sense. But A's data is encrypted and that changes everything. Let's think this through.

Let's say A compresses its data and then encrypts it. And let's say B could somehow decompress the data from A without decrypting it; it can't, but let's suppose. B now has A's decompressed, but encrypted, data. It has, effectively, random bits.

In order to do incremental backups the latest versions of files from A must have something in common with their older versions. But even the smallest change to the unencrypted data will completely change the encrypted data. There are no increments. Each encrypted version of a file from A has nothing in common with the previous version, else you'd be able to extract information. B can't do incremental backups on encrypted data whether the original was compressed or not.

Similarly to do data recovery B must have some sort of pattern to look for. Encrypted data is effectively random. B cannot do data recovery on encrypted data, there's no pattern to look for.

So, from B's perspective, compressed and encrypted data from A is the same as encrypted data from A. It's just smaller.

The integrity of your backup can be handled by conventional means: multiple redundant backups, RAID, off-site backups, and so on. As for incremental backups, I don't think it's possible here. Fortunately that's just about saving disk space. Consider off-loading older versions to cheaper, slower storage.


UPDATE As Barmar mentioned in the comment, A could have to slice up its data into files in such a way so that small logical changes only touched a corresponding number of small files. Then B would dutifully backup the new versions of just those files.

For example, if you had a database of resources rather than having them in one big file A could split them up into one resource per file. The resource reference would be obfuscated as a checksum. The checksum would either be calculated or stored in an index. For example, data for thing 123 might be stored as c27b4225f9ed9b196e307b97f07f04e869d954b9.

stuff/
  objects/
    00/9df0b6c69370f17f4abb0eccf335048497dac3
    97/d5b7466a5eb62e8309d8df06e76d0a7496ed26
    af/d5a4ca06ef2f3c8550a3d2ff923a73dd07b17a
    c2/7b4225f9ed9b196e307b97f07f04e869d954b9

You might recognize this as being very similar to a Git repository.

If A makes a change to a single resource it only changes one (hopefully small) file. B only has to store a new version of a single file.

This does mean B can see what resources are changing, how often, and with what other resources. Even with the obfuscated filenames, this can provide some clues to what's in them. I believe this is a necessary information leak if you want incremental backups; there has to be some boundries to the encrypted data for B to use.

You say you want to decompress the data coming from A so B can do incremental backups and recovery. Were A's data not encrypted this would make perfect sense. But A's data is encrypted and that changes everything. Let's think this through.

Let's say A compresses its data and then encrypts it. And let's say B could somehow decompress the data from A without decrypting it; it can't, but let's suppose. B now has A's decompressed, but encrypted, data. It has, effectively, random bits.

In order to do incremental backups the latest versions of files from A must have something in common with their older versions. But even the smallest change to the unencrypted data will completely change the encrypted data. There are no increments. Each encrypted version of a file from A has nothing in common with the previous version, else you'd be able to extract information. B can't do incremental backups on encrypted data whether the original was compressed or not.

Similarly to do data recovery B must have some sort of pattern to look for. Encrypted data is effectively random. B cannot do data recovery on encrypted data, there's no pattern to look for.

So, from B's perspective, compressed and encrypted data from A is the same as encrypted data from A. It's just smaller.

The integrity of your backup can be handled by conventional means: multiple redundant backups, RAID, off-site backups, and so on. As for incremental backups, I don't think it's possible here. Fortunately that's just about saving disk space. Consider off-loading older versions to cheaper, slower storage.

You say you want to decompress the data coming from A so B can do incremental backups and recovery. Were A's data not encrypted this would make perfect sense. But A's data is encrypted and that changes everything. Let's think this through.

Let's say A compresses its data and then encrypts it. And let's say B could somehow decompress the data from A without decrypting it; it can't, but let's suppose. B now has A's decompressed, but encrypted, data. It has, effectively, random bits.

In order to do incremental backups the latest versions of files from A must have something in common with their older versions. But even the smallest change to the unencrypted data will completely change the encrypted data. There are no increments. Each encrypted version of a file from A has nothing in common with the previous version, else you'd be able to extract information. B can't do incremental backups on encrypted data whether the original was compressed or not.

Similarly to do data recovery B must have some sort of pattern to look for. Encrypted data is effectively random. B cannot do data recovery on encrypted data, there's no pattern to look for.

So, from B's perspective, compressed and encrypted data from A is the same as encrypted data from A. It's just smaller.

The integrity of your backup can be handled by conventional means: multiple redundant backups, RAID, off-site backups, and so on. As for incremental backups, I don't think it's possible here. Fortunately that's just about saving disk space. Consider off-loading older versions to cheaper, slower storage.


UPDATE As Barmar mentioned in the comment, A could have to slice up its data into files in such a way so that small logical changes only touched a corresponding number of small files. Then B would dutifully backup the new versions of just those files.

For example, if you had a database of resources rather than having them in one big file A could split them up into one resource per file. The resource reference would be obfuscated as a checksum. The checksum would either be calculated or stored in an index. For example, data for thing 123 might be stored as c27b4225f9ed9b196e307b97f07f04e869d954b9.

stuff/
  objects/
    00/9df0b6c69370f17f4abb0eccf335048497dac3
    97/d5b7466a5eb62e8309d8df06e76d0a7496ed26
    af/d5a4ca06ef2f3c8550a3d2ff923a73dd07b17a
    c2/7b4225f9ed9b196e307b97f07f04e869d954b9

You might recognize this as being very similar to a Git repository.

If A makes a change to a single resource it only changes one (hopefully small) file. B only has to store a new version of a single file.

This does mean B can see what resources are changing, how often, and with what other resources. Even with the obfuscated filenames, this can provide some clues to what's in them. I believe this is a necessary information leak if you want incremental backups; there has to be some boundries to the encrypted data for B to use.

Source Link
Schwern
  • 461
  • 4
  • 9

You say you want to decompress the data coming from A so B can do incremental backups and recovery. Were A's data not encrypted this would make perfect sense. But A's data is encrypted and that changes everything. Let's think this through.

Let's say A compresses its data and then encrypts it. And let's say B could somehow decompress the data from A without decrypting it; it can't, but let's suppose. B now has A's decompressed, but encrypted, data. It has, effectively, random bits.

In order to do incremental backups the latest versions of files from A must have something in common with their older versions. But even the smallest change to the unencrypted data will completely change the encrypted data. There are no increments. Each encrypted version of a file from A has nothing in common with the previous version, else you'd be able to extract information. B can't do incremental backups on encrypted data whether the original was compressed or not.

Similarly to do data recovery B must have some sort of pattern to look for. Encrypted data is effectively random. B cannot do data recovery on encrypted data, there's no pattern to look for.

So, from B's perspective, compressed and encrypted data from A is the same as encrypted data from A. It's just smaller.

The integrity of your backup can be handled by conventional means: multiple redundant backups, RAID, off-site backups, and so on. As for incremental backups, I don't think it's possible here. Fortunately that's just about saving disk space. Consider off-loading older versions to cheaper, slower storage.