Is there a standard for OpenSSL-interoperable AES encryption?

Question

Many AES-encrypted things (files, strings, database entries, etc.) start with "Salted__" ("U2FsdGVkX1" in base64). I hear it's some sort of OpenSSL interoperability thing: a b c.

Is there some standard reference somewhere (perhaps an RFC?) that explains how such OpenSSL-inter-operable AES-encrypted things are produced and later decrypted?

Ideally, an answer would link to a standard reference for the entire process, or perhaps a brief summary list of steps with a link to a standard reference for each step, something like:

To decrypt such a thing beginning with "U2FsdGVkX1" and with a known password,

• first do base64 decoding -- see Wikipedia: base64. The result will start with "Salted__". Be careful not to use C strings, because the result may include several 0x00 bytes.
• ...
• (I guess something about salting and the IV goes here?)
• (I guess something about CBC or CTR goes here?) -- see Wikipedia: block cipher modes of operation.
• (Perhaps something about message authentication goes here?)
• ...
• AES-decrypt each block with decrypt_one_AES_block( key, block_of_128_bits ) -- see Wikipedia: AES article and A Stick Figure Guide to the Advanced Encryption Standard (AES).
• save the result of each decrypt_one_AES_block(), concatenate them all together, and that's your plaintext. If the original was human-readable text or an HTML file, it may be OK to store the result in a C string; but other kinds of things may include several 0x00 bytes incompatible with C strings.

It may sound like I'm planning to write an implementation myself. I reassure you that I plan to use one of several available libraries -- it's just that when reviewing the libraries, I'd like to know what they should be doing. What should OpenSSL libraries be doing?

Is there a standard for OpenSSL-interoperable AES encryption?

Answer 1

Using the following openssl command as a basis for this answer:

echo -n 'Hello World!' | openssl enc aes-256-cbc -e -a -salt -pbkdf2 -iter 10000 

This command encrypts the plaintext 'Hello World!' using aes-256-cbc. The key is generated using pbkdf2 using the password and a random salt, with 10,000 iterations of sha256 hashing. When prompted for the password, I entered the password, 'p4$$w0rd'. The ciphertext output produced by the command was:

U2FsdGVkX1/Kf8Yo6JjBh+qELWhirAXr78+bbPQjlxE=

Now, to answer the question posed, 'To decrypt such a thing beginning with "U2FsdGVkX1" and with a known password':

Do the following:

1. base64-decode the output from openssl, and utf-8 decode the password, so that we have the underlying bytes for both of these.
2. The salt is bytes 8-15 of the base64-decoded openssl output.
3. Derive a 48-byte key using pbkdf2 given the password bytes and salt with 10,000 iterations of sha256 hashing.
4. The key is bytes 0-31 of the derived key, the iv is bytes 32-47 of the derived key.
5. The ciphertext is bytes 16 through the end of the base64-decoded openssl output.
6. Decrypt the ciphertext using aes-256-cbc, given key, iv, and ciphertext.
7. Remove PKCS#7 padding from plaintext. The last byte of plaintext indicates the number of padding bytes appended to the end of the plaintext. This is the number of bytes to be removed.

Below is a python3 implementation of the above process:

import base64
import hashlib
from Crypto.Cipher import AES       #requires pycrypto

#inputs
openssloutputb64='U2FsdGVkX1/Kf8Yo6JjBh+qELWhirAXr78+bbPQjlxE='
password='p4$$w0rd'
pbkdf2iterations=10000

#convert inputs to bytes
openssloutputbytes=base64.b64decode(openssloutputb64)
passwordbytes=password.encode('utf-8')

#salt is bytes 8 through 15 of openssloutputbytes
salt=openssloutputbytes[8:16]

#derive a 48-byte key using pbkdf2 given the password and salt with 10,000 iterations of sha256 hashing
derivedkey=hashlib.pbkdf2_hmac('sha256', passwordbytes, salt, pbkdf2iterations, 48)

#key is bytes 0-31 of derivedkey, iv is bytes 32-47 of derivedkey 
key=derivedkey[0:32]
iv=derivedkey[32:48]

#ciphertext is bytes 16-end of openssloutputbytes
ciphertext=openssloutputbytes[16:]

#decrypt ciphertext using aes-cbc, given key, iv, and ciphertext
decryptor=AES.new(key, AES.MODE_CBC, iv)
plaintext=decryptor.decrypt(ciphertext)

#remove PKCS#7 padding. 
#Last byte of plaintext indicates the number of padding bytes appended to end of plaintext.  This is the number of bytes to be removed.
plaintext = plaintext[:-plaintext[-1]]

#output results
print('openssloutputb64:', openssloutputb64)
print('password:', password)
print('salt:', salt.hex())
print ('key:', key.hex())
print ('iv:', iv.hex())
print ('ciphertext:', ciphertext.hex())
print ('plaintext:', plaintext.decode('utf-8'))

As expected, the above python3 script produces the following:

openssloutputb64: U2FsdGVkX1/Kf8Yo6JjBh+qELWhirAXr78+bbPQjlxE=
password: p4$$w0rd
salt: ca7fc628e898c187
key:  444ab886d5721fc87e58f86f3e7734659007bea7fbe790541d9e73c481d9d983
iv:  7f4597a18096715d7f9830f0125be8fd
ciphertext:  ea842d6862ac05ebefcf9b6cf4239711
plaintext:  Hello World!

Note: An equivalent/compatible implementation in javascript (using the web crypto api) can be found at https://github.com/meixler/web-browser-based-file-encryption-decryption.

(Added 12/25/2020) This answer applies to openssl v1.1.1, which supports the -pbkdf2 option. Older versions of openssl (or newer versions of openssl with the -md md5 option) used a weak key derivation method based on the md5 hash function, as explained by dave_thompson_085 in his answer below and Thomas Pornin in his answer here. For details on how this key derivation method is implemented, see https://security.stackexchange.com/questions/29106/openssl-recover-key-and-iv-by-passphrase/242567#242567

Answer 2

Since this is still open and the issue keeps coming up:

TLDR: There are lots of things in OpenSSL that implement standards including AES, but the key derivation part of enc is partly nonstandard (at least by default)

First, OpenSSL has several commandline operations it calls commands (although they usually aren't separate programs, as typical commands are on Unix), and a whole range of library calls, that do encryption and decryption with numerous symmetric algorithms of which AES is only one as well as several asymmetric algorithms. Major ones are PKCS#7/CMS and semantically equivalent S/MIME; PKCS#8/rfc5208 and PKCS#12/rfc7292; and the SSL/TLS protocols for which it is named.

The file format with Salted__ is used by one specific OpenSSL command enc. Although if the commandline executable is given a cipher name instead of a command it silently translates to enc -- for example openssl aes-128-cbc ... becomes openssl enc -aes-128-cbc ... -- so people often perceive and describe this as 'OpenSSL AES-128-CBC encryption' as if it were the only AES-128-CBC encryption OpenSSL does.

enc can encrypt and decrypt files (including anything the OS can provide as standard input and output, such as a pipe) with a wide range of options in (mostly) two dimensions:

• the symmetric cipher algorithm and mode, and padding. OpenSSL library provides and enc can use several algorithms, which may vary depending on build but currently default to Blowfish CAST DES DES-EDE(3) (usually called TripleDES TDES or TDEA) IDEA RC2 RC4 Camellia SEED and AES; I haven't bothered linking all the standards. Some algorithms, especially AES, have options for key size. All except RC4 are block ciphers and must be used with a mode such as CBC ECB or CTR; although algorithms and modes are actually defined by separate standards, enc (and the EVP API) combines them in a single name like aes-128-cbc above. (added) A list of the algorithm-and-modes is available through version 1.0.2 from openssl list-cipher-commands or just openssl enc -?, or version 1.1.0 from openssl enc -ciphers. Do not use openssl ciphers which shows something quite different: the ciphersuites available for SSL/TLS protocol communication.

  Some block modes (CBC and ECB) require data be a multiple of the cipher's block size; since in general real data doesn't do this, it is necessary to add padding when encrypting and remove it when decrypting. By default OpenSSL uses the padding scheme defined by PKCS#7 which extends a scheme defined in PKCS#5, and therefore is usually still called PKCS#5 or just PKCS5 padding. If you specify -nopad this is not done, and (for these modes) encrypting (or decrypting) wrong-size data gives an error.

• all symmetric ciphers require a key, and some modes require an Initialization Vector (IV). enc can use a 'raw' key and if applicable IV specified on the commandline, with -K (uppercase!) and -iv and in hexadecimal. However its default and usual mode is to do Password Based Key Derivation (combined with encryption to give Password-Based Encryption, PBE), and this in turn has several options.

  The password can be entered several ways, using legacy option -k (lowercase) or -kfile or newer option -pass or prompted for and typed with no terminal echo (if possible).

  The newer, preferred and default option is salted key derivation, which strengthens the derivation (but see below) by using 8-byte random salt. It is also possible to specify a specific salt with -S in hex, or disable salting with -nosalt. If salt is used, random or not, a very simple header is added to the ciphertext consisting of the ASCII characters Salted__ and the 8 bytes of salt. (Standardized PBEs like PKCS#12 also convey the salt, but use much more extensive and flexible formats.)

  The key derivation process is based on PBKDF1 from PKCS#5/rfc2898 but modified, and is implemented by the API function EVP_BytesToKey whose man page should be available in any Unix system with OpenSSL installed and on the website. A bit confusingly, and inconsistent with more recent best practice, this 'key derivation' actually derives both the key and IV if applicable. To lay out the relationship exactly:

  • PBKDF1 uses a choice of hash from MD2 MD5 SHA1, and an iteration count, and (required) salt. It initially forms password || salt and hashes it, then hashes the first hash, then hashes the second hash, iterating to a total of count times. PKCS5 PBES1 uses PBKDF1 to produce both key and IV -- but only for RC2 and (single) DES, neither of which is acceptable today.

  • EVP_BytesToKey uses any hash supported by EVP, which may depend on build options but currently defaults to MD4 MD5 MDC2 RIPEMD RIPEMD160 SHA1 and the original SHA2 family (but not the later '512 slash' additions), plus iteration count and optional salt. It does one block like PBKDF1 as H^count (password||salt) with the salt omitted if not used, but if the required 'key material' is more than one block, it then does additional blocks as H^count (prevblock||password||salt) ditto. Whether the EVP_BytesToKey result is used for IV depends on the caller.

  • enc uses EVP_BytesToKey with salt by default (but see options above); (edit) hash MD5 by default through 1.0.2 or SHA256 by default in 1.1.0 released 2016-08 but you can specify otherwise with the option -md which was only documented from 1.0.0 but actually available before; and iteration count ONE. Thus the key and IV if applicable used by enc before 1.1.0 defaults to

      key[||IV] = b0=md5(t=pw[||salt]) || b1=md5(b0||t) || b2=md5(b1||t) ...
    

  This is a poor PBKDF. Because all the available hashes (including default MD5 or SHA256) are fast and this PBKDF 'iterates' only once, an attacker can quickly try large numbers of possible passwords to find yours, unless your password contains enough entropy without any significant 'strengthening' -- and most people don't choose, and even if given can't remember, such strong passwords. But we are stuck with it for backward compatibility.

Thus for examples:

openssl rc4-128 [-e|-d] -k sekrit -nosalt 
# uses RC4 with 128-bit key (RC4 is a stream cipher and uses no IV)
# derived using no salt, MD5 (unless 1.1.0), and count 1 from 'sekrit'

openssl aes-256-cbc [-e|-d] -k sekrit -md sha1
# uses AES in CBC mode with 256-bit key and 128-bit IV 
# (both) derived using random salt, SHA1 and count 1 from 'sekrit'
# and adds the Salted__ header with the salt to the ciphertext

And finally, if you specify -base64 or the abbreviation -a, the ciphertext (including header if any) is encoded to base64 on encryption, and decoded from base64 on decryption; this is fairly commonly needed because some applications, systems, or protocols cannot handle the arbitrary (quasi-random) binary bytes needed for ciphertext. (edit) Similarly if you specify -z plaintext is compressed before encryption or decompressed after decryption, but only if OpenSSL was built with compression (zlib), and after the CRIME and BREACH attacks some builders or packagers disable compression entirely. These options are independent of the cipher and key/PBKDF, and in fact you can use them alone to only base64 encode/decode or zlib compress/decompress without doing encryption or decryption at all.

Base64 is used with slight variations in many standards but OpenSSL mostly follows the original PEM in 4.3.2.4 (no internal hyperlink because 1993 was before WWW became popular). Two caveats: base64 decoding silently ignores part and sometimes all of a line with a non-base64 character; this is useful when OpenSSL uses it internally to read PEM-format files, but may not be so in other applications. And before 1.1.0, it silently ignores lines longer than the 76-character limit specified in MIME unless you add -A (uppercase). zlib compression is related to, but not quite the same as, gzip.

Inter-version compatibility: (added) using password-based encryption and decryption depends critically on using the same parameters and especially the same hash in the PBKDF for both operations. As noted above, the default hash used by the enc command changed in 1.1.0 to SHA256 versus MD5 in lower versions. Thus data encrypted using the default hash in lower versions won't decrypt with the default in 1.1.0 or vice versa.

• if data was encrypted with the default (MD5) on an older version, decrypt in 1.1.0 (and presumably higher when released) with -md md5

• if data was encrypted with the default (SHA256) on 1.1.0 (or presumably higher), decrypt in an older version (at least back to 0.9.8, possibly before) with -md sha256

• to proactively prevent a problem, specify -md consistently for both encryption and decryption even when it's redundant

UPDATE: OpenSSL 1.1.1 (2018-09) adds an option for -pbkdf2 -- with default iteration count 10000 -- and an option -iter to change that count -- and gives a warning when you don't use these i.e. when you default to the old scheme, described above. This is the standard PBKDF2 from PKCS5v2/RFC2898, although 10k is rather low (but much better than 1!); even my aging laptop with a Celeron can do 100-200k without noticeable delay. Compare
Can I dynamically calculate an appropriate number of iterations for PBKDF2 based on the system time, rather than using a fixed value?
https://security.stackexchange.com/questions/110084/parameters-for-pbkdf2-for-password-hashing/
https://security.stackexchange.com/questions/3959/recommended-of-iterations-when-using-pkbdf2-sha256 .
But the iteration count isn't recorded in the file, so you must either remember it (which makes varying it inconvenient) or store it elsewhere such as the filename or other metadata like an NTFS alternate stream. Also, it still uses the traditional enc behavior of deriving both the key and IV (if needed), not choosing the IV by a separate random process, as the encryption scheme PBES2 in PKCS5v2 does.

Answer 3

There's no standard for it, it's a proprietary format that OpenSSL invented. So it's interoperable with every other version of OpenSSL out there, but nothing else.

Answer 4

Going off of @mti2935 's answer, here's the Kotlin implementation for decrypting a string value that was encrypted using OpenSSL v1.1.1:

package io.matthewnelson.crypto

import java.util.*
import javax.crypto.Cipher
import javax.crypto.SecretKeyFactory
import javax.crypto.spec.IvParameterSpec
import javax.crypto.spec.PBEKeySpec
import javax.crypto.spec.SecretKeySpec

/**
 * echo "Hello World!" | openssl aes-256-cbc -e -a -p -salt -pbkdf2 -iter 15739 -k qk4aX-EfMUa-g4HdF-fjfkU-bbLNx-15739
 *
 * Terminal output:
 *   salt=CC73B7D29FE59CE1
 *   key=31706F84185EA4B5E8E040F2C813F79722F22996B48B82FF98174F887A9B9993
 *   iv =1420310D41FD7F48E5D8722B9AC1C8DD
 *   U2FsdGVkX1/Mc7fSn+Wc4XLwDsmLdR8O7K3bFPpCglA=
 * */
fun main() {
    val returned = CryptoUtils().decrypt_OpenSSL_AES256CBC_String(
        password = "qk4aX-EfMUa-g4HdF-fjfkU-bbLNx-15739",
        hashIterations = 15739,
        encryptedString = "U2FsdGVkX1/Mc7fSn+Wc4XLwDsmLdR8O7K3bFPpCglA="
    )
    println(returned)
}

class CryptoUtils {

    fun decrypt_OpenSSL_AES256CBC_String(
        password: String,
        hashIterations: Int,
        encryptedString: String
    ): String {
        val encryptedBytes = Base64.getDecoder().decode(encryptedString)

        // Salt is bytes 8 - 15
        val salt = encryptedBytes.copyOfRange(8, 16)
//        println("Salt: ${salt.joinToString("") { "%02X".format(it) }}")

        // Derive 48 byte key
        val keySpec = PBEKeySpec(password.toCharArray(), salt, hashIterations, 48 * 8)
        val keyFactory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA256")
        val secretKey = keyFactory.generateSecret(keySpec)

        // Decryption Key is bytes 0 - 31 of the derived key
        val key = secretKey.encoded.copyOfRange(0, 32)
//        println("Key: ${key.joinToString("") { "%02X".format(it) }}")

        // Input Vector is bytes 32 - 47 of the derived key
        val iv = secretKey.encoded.copyOfRange(32, 48)
//        println("IV: ${iv.joinToString("") { "%02X".format(it) }}")

        // Cipher Text is bytes 16 - end of the encrypted bytes
        val cipherText = encryptedBytes.copyOfRange(16, encryptedBytes.lastIndex + 1)

        // Decrypt the Cipher Text and manually remove padding after
        val cipher = Cipher.getInstance("AES/CBC/NoPadding")
        cipher.init(Cipher.DECRYPT_MODE, SecretKeySpec(key, "AES"), IvParameterSpec(iv))
        val decrypted = cipher.doFinal(cipherText)
//        println("Decrypted: ${decrypted.joinToString("") { "%02X".format(it) }}")

        // Last byte of the decrypted text is the number of padding bytes needed to remove
        val plaintext = decrypted.copyOfRange(0, decrypted.lastIndex + 1 - decrypted.last().toInt())

        return plaintext.toString(Charsets.UTF_8)
    }
}