I'd like to encrypt the contents of a log file that my application writes. The log system has the following characteristics:
- The file format is binary.
- Log files are left open while the application is still running.
- New data is appended as a log entry is made.
- Log entries have varying lengths, I cannot align that with some cipher block size.
- When the application is restarted, it must read an existing log file, understand its contents and decide whether to append new items to this file or start a new file. This requires symmetric encryption.
- The encryption key must be embedded in the application and the receiver of the log file must know it. So I guess it's a static key, at least for that application.
- The decrypted data must not be any longer than the original plaintext, or the log viewer will try to make another complete log item from the additional bytes.
Now this won't allow me to use a super-secure encryption I guess. Somebody who can extract the static key from the application (together with the algorithm used) should be able to read the log file. But then again, the log file should only be encrypted because it may contain information that can reveal internal names that should be obfuscated and encoded in the application. Reading the log file should not help in de-obfuscating the program.
I've used AES in CBC mode to encrypt "secure" configuration files with a static application password that generates the IV and key through Rfc2898DeriveBytes (forgot the name, something like PBKFD...). Together with a random salt that is prepended to the file. But these files are always read and written in one piece. It handles padding well but I'm not yet sure how that works.
It is entirely unclear to me how I should append to a padded encrypted file at a later time. Could I just seek to the last block and overwrite it somehow? (I'm using .NET for my application, if somebody knows what I can do there.)
Then I've read about stream ciphers that process one byte at a time. I think I could stop writing at any point and continue from that point on. But they seem to be generally insecure up to the point that it's useless to encrypt the data at all.
In .NET, I would enclose my file stream with an instance of some encryption/decryption stream. I don't know whether I can get sequential read and write access to the encrypted file with this. I need to read an existing file until its end, then continue only writing. Does that work with encrypted files at all?
So how should I handle encrypting my log files? Are there certain algorithms or modes or ways that are good for this scenario?
Or should I just XOR my log file with a static key, like we used to do it as kids and call it "encryption"...
Some more details, in reply to the comments:
- Log files are only appended to, and eventually deleted in entirety.
- Log entries are binary, incl. NUL bytes. An int32 value takes exactly 4 bytes, for instance.
- A single log entry is technically limited to 256 MiB, but rarely ever nearly as large. (The file format spec is here.)
- Partial encryption would be sufficient, basically only the string entries need encryption, not numbers or the structure itself. (I didn't even think of that solution yet!)
- The log viewer can live-read the log files as they are written. But if there is data after the last complete log item, it must be a valid and complete log item. If it's not complete, the reader will wait for it to become complete. (You could flush each single byte to the file and the log item would appear after the last byte written.) Random noise at the end of the file will break the reading.
The live reading mode is especially interesting considering the padding. While I'm writing to a CryptoStream, the stream must buffer the data of at least one block size. If I close the stream before a block is completed, padding should be appended. If I want to flush the stream but keep it open, the padding would probably not be removed, which breaks that block.
Also, I found that with .NET CryptoStreams, I can either read a non-seekable file or write to it, not both. And I cannot seek in the stream. So appending to an existing file would probably not be an option anyway.
So maybe what I need is not possible with block ciphers in block mode.
But if we find a solution that encrypts only the separate string records (which are referenced from other structures by their file offset, so they're reusable), the problem would narrow down to encrypt separate (mostly short) strings, one at a time, in their own record bounds. I'd probably prefer an algorithm with a small block size to keep the padding overhead small. Is that a different question for this platform? (New problems would be generating the IV and maintaining it through all encrypted string records...)