Problem Overview

I want to securely store log files so the contents are secret, and they can't be modified without detection.

The files will be encrypted using authenticated encryption (AES in GCM mode), with a random IV and symmetric key for each file. The symmetric key will be encrypted using the public part of an RSA key pair. Both the IV and encrypred symmetric key will be included in the additional authenticated data.

This gives me confidentiality, integrity and authenticity - but only for each individual log file.

For example, let's say I have log files 2013-01-01.log, 2013-01-05.log and 2013-02-09.log - an attacker could delete 2013-01-05.log without detection.

I've come up with 2 possible solutions.

Possible Solution 1

The program could maintain an encrypted (and possible RSA-signed) 'counter file', which would contain a sequence number that would be incremented every time we write a new log file. The sequence number would become part of the log filename, and would also be included in the additional authenticated data. We could therefore detect any 'gaps' from missing files.

Possible Solution 2

The program could maintain an encrypted (and possible RSA-signed) 'database file', which would contain the filenames of all previously written log files. We could therefore detect any 'gaps' from missing files.

The Question

I'd like feedback on my 2 possible solutions - do they work, do they need some changed, have I missed anything?

Or are there better solutions to my problem?

  • 1
    $\begingroup$ Why not just use HMAC(key, filename || contents)? There's no need for a unique key for every file. There's no need for encryption. You can also use something like chattr to set the immutable flag, which can only be removed on reboot. $\endgroup$ Apr 24, 2013 at 16:49
  • 1
    $\begingroup$ @rath the IV isn't included in the ciphertext, it's in the additional authenticated data (i.e. in plaintext). Yes, there would be one unique symmetric key per file. $\endgroup$
    – MurrayA
    Apr 24, 2013 at 16:50
  • 2
    $\begingroup$ @StephenTouset how is that any better that just using authenticated encryption? How does it allow for detecting if someone deletes a file? $\endgroup$
    – MurrayA
    Apr 24, 2013 at 16:52
  • 1
    $\begingroup$ So far, none of the proposed methods give any integrity protection after a compromise of the logger. $\endgroup$
    – user991
    Apr 24, 2013 at 17:25
  • 1
    $\begingroup$ @RickyDemer do you have any suggestions? $\endgroup$
    – MurrayA
    Apr 24, 2013 at 17:29

5 Answers 5


"Efficient, Compromise Resilient and Append-only
Cryptographic Schemes for Secure Audit Logging" ​ (PDF)
gives a publicly verifiable approach that allows fine-grained verification,
but it is in the Random Oracle Model.

The Simple Method:
The verifier and logger start with a seed for a
forward-secure pseudo-random number generator.
To denote a valid ending of a log, put the string of
the next $b$ bits of the PRNG's output into the log.
To add a log entry, get the next $\:b+k\:$ bits of the PRNG's output,
put into the log the encryption of the log entry and the mac of that
ciphertext using the last $k$ of the $\:b+k\:$ bits of PRNG output as the
mac key, then erase those $\:b+k\:$ bits and the previous PRNG state.

  • $\begingroup$ An interesting paper that does seem to provide a solution. While I get the basic principles behind it, it's going to take a bit of time before I fully understand it though :) $\endgroup$
    – MurrayA
    Apr 25, 2013 at 8:15
  • $\begingroup$ Updated working link. $\endgroup$
    – Tom Pohl
    Oct 16, 2020 at 8:03

Take a printer, and have the log file come out of the machine on paper. Ensure fire doesn't exist near the paper. Anything else will not work: if an attacker can wind back time log files can die and you cannot tell. All techniques for assuring time cannot be run backwards amount to doing this in some form, perhaps by sending data to another computer. But if you can do that, just send the log files and the attacker cannot even delete them!

  • 1
    $\begingroup$ They actually do sell WORM drives which are the more practical equivalent. $\endgroup$ Apr 27, 2013 at 18:49
  • 1
    $\begingroup$ OK, that is more pratical :) Although I believe such devices are very expensive $\endgroup$
    – MurrayA
    Apr 27, 2013 at 19:46
  • $\begingroup$ Blu-ray writers are cheap (50 GBP), as are 25GB write once discs (1-2 GBP each in small quantities). $\endgroup$
    – Alex
    Jul 6, 2018 at 10:33

Edit: You've clarified in the comments that confidentiality of the logfile's contents is important.

Given an AEAD function $C = E_k(iv, plaintext, aad)$, a safe construct is

$$ C = E_k(iv, contents, filename). $$

There is no need to include either the key or the IV in the additional authenticated data, and I would recommend against doing so. It is plausible that a particular AEAD scheme could leak the contents of the authenticated data, since this is not generally a design requirement.

This scheme will allow you to detect manipulation of the contents of a file, file renaming, and file deletion. You can detect logfile deletion simply by the lack of presence of a file for a particular date. If you have no log data for a date, simply generate an empty file and encrypt it with this scheme; the filename's inclusion in the authenticated data will prevent an attacker from being able to "replay" an encrypted empty file for other dates.

You may additionally want to use something like chattr to set the system-wide immutable flag for your logfiles. This will prevent any user from being able to modify or delete files, without changing the flag as root and rebooting the system first. Similarly, you should consider setting the append-only flag for live logfiles that are still being written to. This will help prevent tampering with them before they can be permanently archived.

Finally, permanent storage media are great solutions to this problem as well. If your need is great enough, burn the signed logfiles to a DVD-R periodically.

  • $\begingroup$ Changing the permissions on completed files is a good idea $\endgroup$
    – MurrayA
    Apr 24, 2013 at 17:05
  • 1
    $\begingroup$ I mentioned attributes, not permissions. Limiting permissions on sensitive files is simply good hygeine and should go without saying. $\endgroup$ Apr 24, 2013 at 17:09
  • $\begingroup$ I'm using Windows, so there is no immutable flag; the next best thing is to change permissions after the file is closed $\endgroup$
    – MurrayA
    Apr 24, 2013 at 17:10
  • $\begingroup$ You could always run *nix on a remote logging server. This also has the advantage of requiring an attacker to break into two boxes to undetectably compromise one of your services. $\endgroup$ Apr 24, 2013 at 17:14
  • 1
    $\begingroup$ Regarding not including the IV or key in the AAD, if I was doing encrypt-then-MAC rather than AE, isn't it good practice to include the IV in the MAC? So shouldn't at least the IV be included in the AAD? crypto.stackexchange.com/a/224/1254 $\endgroup$
    – MurrayA
    Apr 24, 2013 at 17:18

I'm thinking there's a third potential solution. Each time you close a log file, you could append the name of the next new log file, timestamp it, then sign the log file. When it is time to create a new log file, you would read the previous log file, validate the signature, validate the time stamp, read the new log file name, and create it. You'd kickstart the whole thing by self-signing the first empty log file.

This permits you to decrypt each file on its own, which you probably do frequently for ordinary troubleshooting and maintenance activities. When you need to audit the log files, which is probably a less common annual activity, you would walk the chain of all files ensuring that all signatures are valid and that none are missing.

This isn't a perfect or complete solution, of course. Off the cuff, I think a bad guy could tamper with the system clock, setting it back to the end of two log files ago and delete the most recent log file. But clock tampering might leave other evidence in other log files. You should still export the log files to a separate secured server as soon as you close them.

  • $\begingroup$ I like the ralative simplicity of this. I guess using an external, secure timestamping service would alleviate the problem of bad guys changing the system clock $\endgroup$
    – MurrayA
    Apr 26, 2013 at 6:51

I've been toying around with a solution to this problem using blockchain tech.

Say for every client that wants to make a server request, they have to first record their request on a blockchain. Then that client would submit that same request to your server.

Your server would check that the received request also exists on the blockchain before processing the request.

Responses would work the same way but in reverse. The server would publish a response on the blockchain and sent a normal response to the client. The client could then check to make sure that the response from the server also exists on the blockchain.

Something like this

This is probably way overkill, but it's an interesting concept.

  • $\begingroup$ Blockchains can grow in size according to number of transactions, whereas a simple tally+digest of the last block plus metadata is all that's required in the simpler schemes. $\endgroup$
    – MikeW
    Mar 28, 2019 at 14:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.