I have large data set which has sensitive information. I want to encrypt different parts of it with different keys. There is also need for compression as data is very large (100s of GB that has to be distributed to users that have got permission).

I have been reading about format preserving encryption and would like to use it. I would like to know if its a good idea to use FPE after compression or would that even make sense.

  • $\begingroup$ The main advantage of using FPE over normal encryption is that you can force more "structure" onto the ciphertext. Do you need this structure or is it okay with you if your ciphertext looks like white noise (on a bit level)? $\endgroup$ – SEJPM May 29 '17 at 13:23
  • 1
    $\begingroup$ Format preserving encryption only makes sense if data is being moved and handled as individual rows. In that case, it avoids the need for a per row IVs. If the data is being moved in bulk a stream cipher makes more sense. One relatively small IV will encrypt as many rows as you want to bundle together. If separate regions of the plain text need different keys then use multiple stream ciphers. Knowing more about how the data is being used/moved would help with answering the question. $\endgroup$ – Richard Thiessen May 29 '17 at 13:30
  • $\begingroup$ I've removed the last part of the question as this is regarded as too broad. You can roll back the edit, but your question may be put on hold if you do. $\endgroup$ – Maarten Bodewes May 29 '17 at 14:28
  • $\begingroup$ This is an XY problem: You ask if your solution is correct, while it would be much better to ask the actual question / goal you want to achieve. As others have pointed out: FPE is simply the wrong tool in the basket. And it has absolutely nothing to do increasing security (which I take is your actual goal from the comment to an answer below). $\endgroup$ – tylo May 29 '17 at 17:36

I have been reading about format preserving encryption and would like to use it.

There are many good reasons for using FPE, but this is certainly not one of them.

It's OK to get excited about a solution but you should not apply that solution regardless if it fits the problem.

This reminds me of a developer I knew that wanted to use each and every pattern he encountered. Pattern driven development: "we have this programming pattern, now which problem do we need fixing?"

I would like to know if its a good idea to use FPE after compression or would that even make sense.

No, it doesn't make sense. Compression generally compressed to bytes, and bytes can be encrypted fine. FPE is mainly used where there is no easy byte or bit conversion possible, e.g. when the output needs to be a number or a specific range or characters. The most common example is the format of credit card numbers.

Besides that, FPE may add quite a bit of overhead; you certainly don't want to process 100s of GB of data with it.

  • $\begingroup$ Got it. I had this idea that I explained incorrectly (my bad). I have data that has some parts of it which are PII. I need to give them better security than the rest. So I thought about first using FPE on this PII data. Then compressing all the data and then using AES256 or something like that. Does this idea make sense? $\endgroup$ – kneelb4darth May 29 '17 at 16:44
  • $\begingroup$ No it does not. Generally speaking, chaining encryption does not increase security - there are a lot of questions with this regard on this site. And with regard to security, current state-of-the-art encryption schemes are almost never the weakest link in your system - it's much more likely that other things go wrong, like key management, access control, etc. $\endgroup$ – tylo May 29 '17 at 17:47
  • $\begingroup$ @kneelb4darth Not really. Maybe you can compress the data slightly more if you use FPE on parts of it (instead of compressing these parts after encryption with a key). But if you encrypt to bytes then you can compress and add a second layer of protection as well. In the end compression / encryption does not require a specific format as input - just bytes will suffice. $\endgroup$ – Maarten Bodewes May 29 '17 at 19:11
  • $\begingroup$ Ok, Can you point me to some resource or explain how to implement stream cipher with multiple keys. I mean, if I break data into parts and then encrypt parts separately, then I will have to send them separately too, making it difficult. I will then have to take care that all parts have reached or not. So how to properly do the stream cipher with multiple keys? $\endgroup$ – kneelb4darth May 29 '17 at 19:14
  • $\begingroup$ Huh? You first encrypt with the first, then with the other. Maybe you need some kind of way of distinguishing the frames. You can do this e.g. by defining a Tag + Length + Value encoding or indeed by specifying an ASN.1 structure (if you're up to that). Sounds like you need a protocol / file specification. That's a bit much to put in the answer though, and I would not have enough info (or time) anyway. $\endgroup$ – Maarten Bodewes May 29 '17 at 19:17

It's actually impossible /pointless to use FPE after compression, for two connected reasons.

A good compression algorithm will eliminate all redundancy from a source. That includes all forms of formatting itself such as white space, commas etc. They will be treated exactly like any other character, so if you have a lot of "Miss ", that might get encoded to one byte value.

Secondly, compression seeks to fill up the available bytes by using all the bits for storage. So if your alphabetic source only used 26 characters, that's utilising 4.7 bits. The remaining 3.3 bits will be filled with other encoded information resulting in all 8 bits being use, ie. a full byte. So your compressed output will simply consist of 99.9% uniformly distributed random looking bytes. There's a little more to it than that, but this trivial explanation should illustrate the point.

Both reasons result in an output file that has no format whatsoever worth preserving. Clearly there will be a little protocol format for the decompressor, but that's algorithmic format and not source format.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.