I'm struggling to implement AES256-GCM on a Linux machine to encrypt files outgoing to another party.

We're usually exchanging files using PKI, but this particular party insists on AES256-GCM and will not allow PKI.

However, I've been struggling to make it work. We'd like to implement it on a Linux machine and to be able to automate encryption. It seems like the other party is okay with doing this manually on a Windows desktop, but we'd like to avoid that possibility.

I tried to do it using openssl, but I get this:

$ openssl aes-256-gcm
AEAD ciphers not supported by the enc utility

When I tried GPG, I get:

$ gpg --version

The other party just suggested using what they have, which is some Windows EXE with BouncyCastle libraries, so I started investigating using BouncyCastle for this, but it's in Java, and I'd like to avoid using Java if possible.

I'd have to add a JDK to that particular server or set up another virtual machine just for the purpose of encrypting these files, which is completely non-optimal. Browsing BouncyCastle's documentation, it's not very helpful, at least I haven't found any real examples, and I must be doing something wrong, because even when I did a full text search looking for a reference to GCM, Galois Counter, or anything similar, I found nothing (do I have the wrong files?).

I figure I might work it out, but we've been given a tight deadline and we can't force them to use PKI. I hate to waste time looking in the wrong places, but I would really want to avoid using the simplistic manual process in Windows if it's at all possible.

Edit: The expected format is IV+ciphertext(plus, I presume, tag). I received a copy of their Java source code for a simple encrypt/decrypt utility and it's very simplistic with no allowances for complex encoding routines or anything, it's just meant to encrypt (or decrypt) a single file or a single zip package of multiple files. That's all.

The worst thing is that I saw references to being able to encrypt using OpenSSL (which is just an interface for enc, after all, so it should also work with gpg, for instance), and the best one I found was this: https://security.stackexchange.com/questions/128883/basic-question-regarding-openssl-and-aes-gcm

I guess I'll try that tomorrow and see if I can get it to work.

  • 2
    $\begingroup$ What format they expect? AES-GCM is just the algorithm, but how will you structure the file? (i.e. simply concatenate the IV, ciphertext and tag?). You'll need to determine that with the other party. Afterwards, you'll just need to use any crypto library. OpenSSL is not just a command line client but also a library. Which languages do you prefer? BouncyCastle does support GCM but I couldn't find a good example with a quick search. $\endgroup$
    – Conrado
    Dec 3, 2019 at 17:54
  • $\begingroup$ They expect to concatenate IV and ciphertext, zipped. I'm not sure about the tag. As for the language, my most preferred would be a binary package available directly in a standard RHEL repository. $\endgroup$
    – Marek
    Dec 3, 2019 at 19:19
  • $\begingroup$ BouncyCastle is implemented in both Java and dot-NET (C#), and a 'Windows EXE' is likely using the latter. That won't work directly on Linux of course; I don't know if it's among the cases wine handles. $\endgroup$ Dec 4, 2019 at 5:24
  • $\begingroup$ I tried using mono to run the C# version and it worked, but only marginally. Whatever the error, mono returns an exit status of 0, so there's no way to automate this except responding to every possible error string in an undocumented EXE file. I'd love to be able to automate this, but I do have a recourse possible in that I can just push it to another team as a manual process since I would have to babysit every execution. $\endgroup$
    – Marek
    Dec 4, 2019 at 12:54
  • $\begingroup$ Err, AES256-GCM is kinda standardised and implementation agnostic. That means it doesn't have to be executed on Linux. The cipher text should be identical on any platform. So if this is a £££ project, simply encrypt on something else that you spin up in a VM. $\endgroup$
    – Paul Uszak
    Dec 6, 2019 at 21:22

3 Answers 3


LibreSSL supports AEAD ciphers, including aes-256-gcm:

$ openssl enc -aes-256-gcm -nosalt -p -in file.in -out file.out
enter aes-256-gcm encryption password:
Verifying - enter aes-256-gcm encryption password:
iv =BDEEA37B93BB989C6C40665B

If you don't mind writing your own software, there are plenty of crypto libraries supporting AES-GCM, such as OpenSSL itself (even if not available from the command line tool). Libsodium also provides AES256-GCM and may be easier to use.

That being said, using a common and trusted tool such as GPG would be way safer than implementing your own software, even if the resulting files are not encrypted using AES-GCM.

  • $\begingroup$ We routinely use PKI for encryption to other parties, but this one insists on AES-GCM and it's non-negotiable. $\endgroup$
    – Marek
    Dec 4, 2019 at 7:00
  • $\begingroup$ Unfortunately I cannot use LibreSSL. I checked and it won't fly, at least not until Red Hat (or Oracle) introduces it into their repositories. FWIW, OpenSSL supports AEAD ciphers but requires the correct version of enc. I saw references to this, but never any actual implementation. $\endgroup$
    – Marek
    Dec 4, 2019 at 20:56
  • $\begingroup$ I got enc: AEAD ciphers not supported for fedora 29 x64 workstation OpenSSL 1.1.1d FIPS 10 Sep 2019 . $\endgroup$
    – Nick Dong
    Mar 7, 2020 at 10:45
  • $\begingroup$ I would not say libressl supports AEAD ciphers, because the MAC check is also not implemented in libressl. That is why you will get the error message "bad decrypt" when you decrypt the cipher text obtained from your command. libressl just does not terminate when you call aes-gcm from the command line. OpenSSL does so because of security concerns, see github.com/openssl/openssl/issues/12220 $\endgroup$ Apr 29, 2021 at 13:19

In the end, it was easiest to just code this in Python. For those interested, here's the bare basics code.

DO NOT USE THIS CODE IN PRODUCTION There's no real file handling, no handling of decryption errors, IV and key are in code and not randomly determined.

Importing relevant libraries:

import os
from cryptography.hazmat.primitives.ciphers.aead import AESGCM

For encryption:

with open('/path/to/file/plaintext', 'rb') as infile:
    data = infile.read()

aad = None
key = '12345678123456781234567812345678'
aesgcm = AESGCM(key.encode('utf-8'))
nonce = '123456781234'
ciphertxt = aesgcm.encrypt(nonce.encode('utf-8'), data, aad)

Next is writing the result. The first 12 bytes is the IV, ciphertext follows:

with open('/path/to/file/encrypted_output', 'wb+') as outfile:

As for decryption:

with open('/path/to/fileciphertext', 'rb') as infile:
    nonce = infile.read(12)
    data = infile.read()

aad = None
key = '12345678123456781234567812345678'
aesgcm = AESGCM(key.encode('utf-8'))
plaintxt = aesgcm.decrypt(nonce, data, aad)

with open('/path/to/output/plaintext', 'wb+') as outfile:

That's the format expected by the other party, so I hope this is enough. I'd be glad to hear about any and all errors in that code above.


You could also check if you're version of OpenSSL supports CCM mode. That's an authenticated mode just like GCM. I tried a version of OpenSSL I happened to have on my machine and it supports CCM, but not GCM.

  • 1
    $\begingroup$ Unfortunately, GCM is the only mode the other party accepts. $\endgroup$
    – Marek
    Dec 4, 2019 at 6:58

Not the answer you're looking for? Browse other questions tagged or ask your own question.