Encryption of log files

Question

I'd like to encrypt the contents of a log file that my application writes. The log system has the following characteristics:

• The file format is binary.
• Log files are left open while the application is still running.
• New data is appended as a log entry is made.
• Log entries have varying lengths, I cannot align that with some cipher block size.
• When the application is restarted, it must read an existing log file, understand its contents and decide whether to append new items to this file or start a new file. This requires symmetric encryption.
• The encryption key must be embedded in the application and the receiver of the log file must know it. So I guess it's a static key, at least for that application.
• The decrypted data must not be any longer than the original plaintext, or the log viewer will try to make another complete log item from the additional bytes.

Now this won't allow me to use a super-secure encryption I guess. Somebody who can extract the static key from the application (together with the algorithm used) should be able to read the log file. But then again, the log file should only be encrypted because it may contain information that can reveal internal names that should be obfuscated and encoded in the application. Reading the log file should not help in de-obfuscating the program.

I've used AES in CBC mode to encrypt "secure" configuration files with a static application password that generates the IV and key through Rfc2898DeriveBytes (forgot the name, something like PBKFD...). Together with a random salt that is prepended to the file. But these files are always read and written in one piece. It handles padding well but I'm not yet sure how that works.

It is entirely unclear to me how I should append to a padded encrypted file at a later time. Could I just seek to the last block and overwrite it somehow? (I'm using .NET for my application, if somebody knows what I can do there.)

Then I've read about stream ciphers that process one byte at a time. I think I could stop writing at any point and continue from that point on. But they seem to be generally insecure up to the point that it's useless to encrypt the data at all.

In .NET, I would enclose my file stream with an instance of some encryption/decryption stream. I don't know whether I can get sequential read and write access to the encrypted file with this. I need to read an existing file until its end, then continue only writing. Does that work with encrypted files at all?

So how should I handle encrypting my log files? Are there certain algorithms or modes or ways that are good for this scenario?

Or should I just XOR my log file with a static key, like we used to do it as kids and call it "encryption"...

Some more details, in reply to the comments:

• Log files are only appended to, and eventually deleted in entirety.
• Log entries are binary, incl. NUL bytes. An int32 value takes exactly 4 bytes, for instance.
• A single log entry is technically limited to 256 MiB, but rarely ever nearly as large. (The file format spec is here.)
• Partial encryption would be sufficient, basically only the string entries need encryption, not numbers or the structure itself. (I didn't even think of that solution yet!)
• The log viewer can live-read the log files as they are written. But if there is data after the last complete log item, it must be a valid and complete log item. If it's not complete, the reader will wait for it to become complete. (You could flush each single byte to the file and the log item would appear after the last byte written.) Random noise at the end of the file will break the reading.

The live reading mode is especially interesting considering the padding. While I'm writing to a CryptoStream, the stream must buffer the data of at least one block size. If I close the stream before a block is completed, padding should be appended. If I want to flush the stream but keep it open, the padding would probably not be removed, which breaks that block.

Also, I found that with .NET CryptoStreams, I can either read a non-seekable file or write to it, not both. And I cannot seek in the stream. So appending to an existing file would probably not be an option anyway.

So maybe what I need is not possible with block ciphers in block mode.

But if we find a solution that encrypts only the separate string records (which are referenced from other structures by their file offset, so they're reusable), the problem would narrow down to encrypt separate (mostly short) strings, one at a time, in their own record bounds. I'd probably prefer an algorithm with a small block size to keep the padding overhead small. Is that a different question for this platform? (New problems would be generating the IV and maintaining it through all encrypted string records...)

Answer 1

Considering that you can never really solve your problem as long as you are running in an untrusted environment, your solution is fine.

To append to an old logfile you need to read the last and second last block, decrypt the last one using the second last one as "IV", remove the padding and start appending. Beginning by overwriting the last block you can now simply append your new encrypted blocks to the file. For encrypting you use the second last (cyphertext) block as "IV" again.

See https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Cipher-block_chaining_.28CBC.29 to understand the principle.

Answer 2

Then I've read about stream ciphers that process one byte at a time. I think I could stop writing at any point and continue from that point on. But they seem to be generally insecure up to the point that it's useless to encrypt the data at all.

Using just a stream cipher won't work. However, stream cipher (with a nonce/IV) + HMAC would. I.e. whenever you stop (or after every write) take the HMAC of the ciphertext so far and write it to another file.

Incrementally calculating an HMAC is efficient. I don't know how the interface looks in .NET, but in python I would do:

hmac.update(log_ctext) # update the running HMAC state
hmac.digest()          # find the current digest while retaining the state

That means when you parse the log file on startup to verify the HMAC, you are already ready to append new data without overwriting anything. You just have to be careful to throw out the whole file if the HMAC doesn't match, and generate a new nonce/IV.

Answer 3

If it's a hard constraint that the length of the encrypted log entry must exactly match the length of the unencrypted log entry, then you want a length-preserving encryption scheme. I recommend you use one of the length-preserving schemes pioneered for disk encryption. They essentially form a tweakable block cipher with a variable block length, so they'll be perfect for encrypting a variable-length log entry. I suggest you use a hash of the previous ciphertext as the tweak for encrypting the current log entry.

For very short log entries, you could also look at format-preserving encryption.

Note that none of these will provide any integrity protection, unless there is some redundancy in log entries that your log reader checks. If there is some redundancy that is checked by the log reader, then the tweakable variable-width-block block ciphers will provide the best possible integrity, given the constraints that encryption cannot increase the length of the record.

If encryption is allowed to increase the length of the record slightly, then I recommend using an authenticated encryption scheme, such as GCM, EAX, etc.