# Tamper-proofing log files

Problem Overview

I want to securely store log files so the contents are secret, and they can't be modified without detection.

The files will be encrypted using authenticated encryption (AES in GCM mode), with a random IV and symmetric key for each file. The symmetric key will be encrypted using the public part of an RSA key pair. Both the IV and encrypred symmetric key will be included in the additional authenticated data.

This gives me confidentiality, integrity and authenticity - but only for each individual log file.

For example, let's say I have log files 2013-01-01.log, 2013-01-05.log and 2013-02-09.log - an attacker could delete 2013-01-05.log without detection.

I've come up with 2 possible solutions.

Possible Solution 1

The program could maintain an encrypted (and possible RSA-signed) 'counter file', which would contain a sequence number that would be incremented every time we write a new log file. The sequence number would become part of the log filename, and would also be included in the additional authenticated data. We could therefore detect any 'gaps' from missing files.

Possible Solution 2

The program could maintain an encrypted (and possible RSA-signed) 'database file', which would contain the filenames of all previously written log files. We could therefore detect any 'gaps' from missing files.

The Question

I'd like feedback on my 2 possible solutions - do they work, do they need some changed, have I missed anything?

Or are there better solutions to my problem?

-
Why not just use HMAC(key, filename || contents)? There's no need for a unique key for every file. There's no need for encryption. You can also use something like chattr to set the immutable flag, which can only be removed on reboot. –  Stephen Touset Apr 24 '13 at 16:49
@rath the IV isn't included in the ciphertext, it's in the additional authenticated data (i.e. in plaintext). Yes, there would be one unique symmetric key per file. –  MurrayA Apr 24 '13 at 16:50
@StephenTouset how is that any better that just using authenticated encryption? How does it allow for detecting if someone deletes a file? –  MurrayA Apr 24 '13 at 16:52
So far, none of the proposed methods give any integrity protection after a compromise of the logger. –  Ricky Demer Apr 24 '13 at 17:25
@RickyDemer do you have any suggestions? –  MurrayA Apr 24 '13 at 17:29

"Efﬁcient, Compromise Resilient and Append-only Cryptographic Schemes for Secure Audit Logging" (PDF) gives a publicly verifiable approach that allows
fine-grained verification, but it is in the Random Oracle Model.

The Simple Method:
The verifier and logger start with a seed for a forward-secure pseudo-random number generator.
To denote a valid ending of a log, put the string of the next $b$ bits of the PRNG's output into the log.
To add a log entry, get the next $\:b+k\:$ bits of the PRNG's output, put into the log the
encryption of the log entry and the mac of that ciphertext using the last $k$ of the $\:b+k\:$ bits
of PRNG output as the mac key, then erase those $\:b+k\:$ bits and the previous PRNG state.

-
An interesting paper that does seem to provide a solution. While I get the basic principles behind it, it's going to take a bit of time before I fully understand it though :) –  MurrayA Apr 25 '13 at 8:15

Edit: You've clarified in the comments that confidentiality of the logfile's contents is important.

Given an AEAD function $C = E_k(iv, plaintext, aad)$, a safe construct is

$$C = E_k(iv, contents, filename).$$

There is no need to include either the key or the IV in the additional authenticated data, and I would recommend against doing so. It is plausible that a particular AEAD scheme could leak the contents of the authenticated data, since this is not generally a design requirement.

This scheme will allow you to detect manipulation of the contents of a file, file renaming, and file deletion. You can detect logfile deletion simply by the lack of presence of a file for a particular date. If you have no log data for a date, simply generate an empty file and encrypt it with this scheme; the filename's inclusion in the authenticated data will prevent an attacker from being able to "replay" an encrypted empty file for other dates.

You may additionally want to use something like chattr to set the system-wide immutable flag for your logfiles. This will prevent any user from being able to modify or delete files, without changing the flag as root and rebooting the system first. Similarly, you should consider setting the append-only flag for live logfiles that are still being written to. This will help prevent tampering with them before they can be permanently archived.

Finally, permanent storage media are great solutions to this problem as well. If your need is great enough, burn the signed logfiles to a DVD-R periodically.

-
Changing the permissions on completed files is a good idea –  MurrayA Apr 24 '13 at 17:05
I mentioned attributes, not permissions. Limiting permissions on sensitive files is simply good hygeine and should go without saying. –  Stephen Touset Apr 24 '13 at 17:09
I'm using Windows, so there is no immutable flag; the next best thing is to change permissions after the file is closed –  MurrayA Apr 24 '13 at 17:10
You could always run *nix on a remote logging server. This also has the advantage of requiring an attacker to break into two boxes to undetectably compromise one of your services. –  Stephen Touset Apr 24 '13 at 17:14
Regarding not including the IV or key in the AAD, if I was doing encrypt-then-MAC rather than AE, isn't it good practice to include the IV in the MAC? So shouldn't at least the IV be included in the AAD? crypto.stackexchange.com/a/224/1254 –  MurrayA Apr 24 '13 at 17:18

I'm thinking there's a third potential solution. Each time you close a log file, you could append the name of the next new log file, timestamp it, then sign the log file. When it is time to create a new log file, you would read the previous log file, validate the signature, validate the time stamp, read the new log file name, and create it. You'd kickstart the whole thing by self-signing the first empty log file.

This permits you to decrypt each file on its own, which you probably do frequently for ordinary troubleshooting and maintenance activities. When you need to audit the log files, which is probably a less common annual activity, you would walk the chain of all files ensuring that all signatures are valid and that none are missing.

This isn't a perfect or complete solution, of course. Off the cuff, I think a bad guy could tamper with the system clock, setting it back to the end of two log files ago and delete the most recent log file. But clock tampering might leave other evidence in other log files. You should still export the log files to a separate secured server as soon as you close them.

-
I like the ralative simplicity of this. I guess using an external, secure timestamping service would alleviate the problem of bad guys changing the system clock –  MurrayA Apr 26 '13 at 6:51

Take a printer, and have the log file come out of the machine on paper. Ensure fire doesn't exist near the paper. Anything else will not work: if an attacker can wind back time log files can die and you cannot tell. All techniques for assuring time cannot be run backwards amount to doing this in some form, perhaps by sending data to another computer. But if you can do that, just send the log files and the attacker cannot even delete them!

-
Yes, very practical, thanks... –  MurrayA Apr 27 '13 at 9:19
They actually do sell WORM drives which are the more practical equivalent. –  Watson Ladd Apr 27 '13 at 18:49
OK, that is more pratical :) Although I believe such devices are very expensive –  MurrayA Apr 27 '13 at 19:46