1
$\begingroup$

Are there any encryption schemes that could be classified as "appendable"?

By appendable I mean being able to add new data to the end of already encrypted data, without needing to decrypt the entire file.

My scenario is the following: I have a "log" file that is appended to by a program. It is not a traditional log file. The authenticity of the data is not of any concern. The files contents should be private, and therefor the program should not be able to decrypt the data without the users private key or some password. One solution to the appendable issue is to encrypt each line separately with a salt or similar. The problem with that is that I want the generated log files to stand the test of time, and not need any custom decryption implementation.

Ideally the decryption should be a single shell command, using only off-the-shelf software.

Another reason for the need of an appendable format is speed. The log files will eventually become very large, multiple hundreds of megabytes unencrypted. New rows might not have time to wait for the decryption and the extra processing is wasteful.

The following is my home made requirements:

  • Write to the end without decrypting

My real requirements:

  • Writing is fast
  • File contents are hidden from the program
  • Decrypt using standard software
  • Secure

What i don't care about:

  • Decryption speed
  • Authenticity

To avoid the classic XY problem: If there is another much simpler way to solve the issue at hand, please say it. And if my solution really is the only way of doing this, and nothing fits the criteria: if we allow major tradeoffs, what would you recommend doing instead?

$\endgroup$
4
  • 1
    $\begingroup$ Stream cipher. e.g. AES-CTR, ChaCha20. $\endgroup$
    – DannyNiu
    Commented May 22 at 10:45
  • $\begingroup$ How would you prevent needing to store a symmetric key in the software then? Isn't it a security issue if the key can be extracted from the binary or from memory? $\endgroup$
    – Treday
    Commented May 22 at 11:45
  • $\begingroup$ I was probably unclear in my question but it should be impossible, even if you have physical access to the machine running the software, to decrypt the file. $\endgroup$
    – Treday
    Commented May 22 at 11:50
  • $\begingroup$ You might use "not quite appendable" data. So forgetting about crypto for a moment, you can append data to a file, but unless it's opened in "APPEND" mode, you can read say the last 100 bytes, then combine those 100 bytes with say further 10 bytes, and overwrite the 100 bytes with 110 bytes. And if you store 100 bytes of state at the end of the file, you might be able to replace the 100 bytes of state with 10 encrypted bytes and another 100 bytes of new state. Obviously you need to check if that doesn't allow breaking your cryptography. $\endgroup$
    – gnasher729
    Commented May 22 at 16:26

1 Answer 1

1
$\begingroup$

This is a function of stream ciphers that are not plaintext aware - which is usually the case. These ciphers have a random access property which allows you to encrypt / decrypt any byte in the stream. In other words, these ciphers are seekable, a different term that is often used.

The way this works kind of stream ciphers create a key stream which is then XORed with the plaintext. So some API's allow you to start encrypting / decrypting at a specific offset. You'd still need to set the nonce of course, unless you have one specific key for each log file in which case the nonce could be kept at an all zero value.


Counter mode is a popular mode of operation that turns a block cipher into a stream cipher. Counter mode encrypts a block representing the counter to create one block of key stream - a detail that is usually hidden by the cipher implementation.

If the implementation does not allow to set the counter or to access the key stream then it is possible to calculate the key stream

  1. calculate the block number from the offset by dividing by the block size;
  2. generate the required counter block by combining the block number and the nonce;
  3. setting the IV to the calculated counter block (the IV is the first counter block);
  4. skipping as many bytes as required to get to the right offset (offset % blocksize).

This is assuming the block size in bytes rather than bits.


There are some issues with this approach though. First of all, if you encrypt all entries together then it won't be easy to search through the entries. One trick is to define a specific size for each entry, or at least a maximum size as that would allow the user to search backwards.

Furthermore, this approach doesn't (yet) allow for any authentication of the log entries or log file. Quite often you want to look for an entry and make sure that it hasn't been altered, especially for audit logs.

Trickiest part for those is to make sure that log entries aren't removed. For that it's best to perform log file management separate from the log file, because otherwise an adversary can simply revert to a previous version of the log file.


Needless to say, there are many protocols and file formats for logging data, including ones that allow for (partial) encryption of log information, so please make sure that you are not reinventing the wheel.

Generally log files just encrypt the confidential data. This can be performed using e.g. openssl to encrypt and base 64 encode the data. On Unix systems this would have a CLI like:

key=$(cat encryption_key.bin)
message="This is a test log entry"
iv=$(openssl rand -hex 16)
encrypted_message=$(echo -n "$message" | openssl enc -aes-256-cbc -a -K $(echo -n "$key" | xxd -p -c 256) -iv $iv)
logger -p local0.info "Encrypted log entry: IV=$iv, $encrypted_message"

Note that you should look into correctly using PBKDF2 if you want to use a password rather than a key.


I don't think there are many protocols that allow for a completely encrypted log file format. If you want that you might want to simply mount an encrypted filesystem such as EncFS, write to a log file and then unmount when the logging is finished. That way you've got reasonably well encrypted data (look into disk encryption to understand what I mean with "reasonable") while offering a transparent interface to the logging process and all the required tooling.

$\endgroup$
2
  • $\begingroup$ Thank you for some leads, but the remaining issue is how to do this in an asymmetric fashion, or in some other way prevent needing to store the symmetric key on disk. The software should only be able to write encrypt, never decrypt previous entries. $\endgroup$
    – Treday
    Commented May 23 at 6:40
  • $\begingroup$ @Treday Use hybrid encryption: you store the symmetric key encrypted with the public key of an asymmetric key pair, while the private key is kept away by the authorized user. $\endgroup$
    – not2savvy
    Commented May 23 at 6:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.