9
$\begingroup$

Many AES-encrypted things (files, strings, database entries, etc.) start with "Salted__" ("U2FsdGVkX1" in base64). I hear it's some sort of OpenSSL interoperability thing: a b c.

Is there some standard reference somewhere (perhaps an RFC?) that explains how such OpenSSL-inter-operable AES-encrypted things are produced and later decrypted?

Ideally, an answer would link to a standard reference for the entire process, or perhaps a brief summary list of steps with a link to a standard reference for each step, something like:

To decrypt such a thing beginning with "U2FsdGVkX1" and with a known password,

  • first do base64 decoding -- see Wikipedia: base64. The result will start with "Salted__". Be careful not to use C strings, because the result may include several 0x00 bytes.
  • ...
  • (I guess something about salting and the IV goes here?)
  • (I guess something about CBC or CTR goes here?) -- see Wikipedia: block cipher modes of operation.
  • (Perhaps something about message authentication goes here?)
  • ...
  • AES-decrypt each block with decrypt_one_AES_block( key, block_of_128_bits ) -- see Wikipedia: AES article and A Stick Figure Guide to the Advanced Encryption Standard (AES).
  • save the result of each decrypt_one_AES_block(), concatenate them all together, and that's your plaintext. If the original was human-readable text or an HTML file, it may be OK to store the result in a C string; but other kinds of things may include several 0x00 bytes incompatible with C strings.

It may sound like I'm planning to write an implementation myself. I reassure you that I plan to use one of several available libraries -- it's just that when reviewing the libraries, I'd like to know what they should be doing. What should OpenSSL libraries be doing?

Is there a standard for OpenSSL-interoperable AES encryption?

$\endgroup$
  • 1
    $\begingroup$ related: "Salting when encrypting?" $\endgroup$ – David Cary Oct 2 '12 at 16:10
  • 1
    $\begingroup$ related: "Wheres the salt on the openssl aes encryption?" $\endgroup$ – David Cary Jan 24 '14 at 3:49
  • 1
    $\begingroup$ A very good question, and I have been programming down this rabbit hole recently as well. @Dave correctly notes that OpenSSL does some weird things that are proprietary, but he is incorrect that "nothing else" is compatible. Today I stumbled upon a nearly complete implementation of 'openssl enc' in Java via: juliusdavies.ca/commons-ssl/pbe.html You can, with the java class org.apache.commons.ssl.OpenSSL, do the equivalent in Java of nearly any command line such as 'openssl enc -aes-128-cbc -iv foo -key bar -base64', or 'openssl enc -aes-128-cbc -pass /foo/bar'. The ciphertext is interoperable $\endgroup$ – Douglas Held Aug 7 '15 at 21:42
5
$\begingroup$

There's no standard for it, it's a proprietary format that OpenSSL invented. So it's interoperable with every other version of OpenSSL out there, but nothing else.

$\endgroup$
  • 2
    $\begingroup$ Slightly more information would be welcome, e.g. a reference and a very short description of what the actual format is. A pointer to CMS containers would also be great. $\endgroup$ – Maarten Bodewes Aug 9 '15 at 10:05
  • $\begingroup$ OpenSSL doesn't use CMS containers, it's a proprietary format. The thing with proprietary formats is that they're, well, proprietary. You'd have to reverse-engineer the OpenSSL source code to figure out what the actual format is. $\endgroup$ – Dave Apr 28 '16 at 16:06
  • $\begingroup$ Ah, so openssl pkcs7 doesn't exist? Note the final question: "Is there a standard for OpenSSL-interoperable AES encryption?" $\endgroup$ – Maarten Bodewes Apr 28 '16 at 16:39
  • 1
    $\begingroup$ You can use PKCS #7/CMS, but the original question was about OpenSSL's base64 format. If it was CMS it'd be either raw binary or denoted with "----- BEGIN PKCS7 MESSAGE -----" or something similar. There is a standard for CMS-based AES encryption (and PGP-based AES encryption), but not for whatever format OpenSSL uses with its base64 format. $\endgroup$ – Dave Apr 28 '16 at 16:49
  • $\begingroup$ It's incorrect to call it proprietary. $\endgroup$ – Melab Sep 10 '18 at 0:22
17
$\begingroup$

Since this is still open and the issue keeps coming up:

TLDR: There are lots of things in OpenSSL that implement standards including AES, but the key derivation part of enc is partly nonstandard

First, OpenSSL has several commandline operations it calls commands (although they usually aren't separate programs, as typical commands are on Unix), and a whole range of library calls, that do encryption and decryption with numerous symmetric algorithms of which AES is only one as well as several asymmetric algorithms. Major ones are PKCS#7/CMS and semantically equivalent S/MIME; PKCS#8/rfc5208 and PKCS#12/rfc7292; and the SSL/TLS protocols for which it is named.

The file format with Salted__ is used by one specific OpenSSL command enc. Although if the commandline executable is given a cipher name instead of a command it silently translates to enc -- for example openssl aes-128-cbc ... becomes openssl enc -aes-128-cbc ... -- so people often perceive and describe this as 'OpenSSL AES-128-CBC encryption' as if it were the only AES-128-CBC encryption OpenSSL does.

enc can encrypt and decrypt files (including anything the OS can provide as standard input and output, such as a pipe) with a wide range of options in (mostly) two dimensions:

  • the symmetric cipher algorithm and mode, and padding. OpenSSL library provides and enc can use several algorithms, which may vary depending on build but currently default to Blowfish CAST DES DES-EDE(3) (usually called TripleDES TDES or TDEA) IDEA RC2 RC4 Camellia SEED and AES; I haven't bothered linking all the standards. Some algorithms, especially AES, have options for key size. All except RC4 are block ciphers and must be used with a mode such as CBC ECB or CTR; although algorithms and modes are actually defined by separate standards, enc (and the EVP API) combines them in a single name like aes-128-cbc above. (added) A list of the algorithm-and-modes is available through version 1.0.2 from openssl list-cipher-commands or just openssl enc -?, or version 1.1.0 from openssl enc -ciphers. Do not use openssl ciphers which shows something quite different: the ciphersuites available for SSL/TLS protocol communication.

    Some block modes (CBC and ECB) require data be a multiple of the cipher's block size; since in general real data doesn't do this, it is necessary to add padding when encrypting and remove it when decrypting. By default OpenSSL uses the padding scheme defined by PKCS#7 which extends a scheme defined in PKCS#5, and therefore is usually still called PKCS#5 or just PKCS5 padding. If you specify -nopad this is not done, and (for these modes) encrypting (or decrypting) wrong-size data gives an error.

  • all symmetric ciphers require a key, and some modes require an Initialization Vector (IV). enc can use a 'raw' key and if applicable IV specified on the commandline, with -K (uppercase!) and -iv and in hexadecimal. However its default and usual mode is to do Password Based Key Derivation (combined with encryption to give Password-Based Encryption, PBE), and this in turn has several options.

    The password can be entered several ways, using legacy option -k (lowercase) or -kfile or newer option -pass or prompted for and typed with no terminal echo (if possible).

    The newer, preferred and default option is salted key derivation, which strengthens the derivation (but see below) by using 8-byte random salt. It is also possible to specify a specific salt with -S in hex, or disable salting with -nosalt. If salt is used, random or not, a very simple header is added to the ciphertext consisting of the ASCII characters Salted__ and the 8 bytes of salt. (Standardized PBEs like PKCS#12 also convey the salt, but use much more extensive and flexible formats.)

    The key derivation process is based on PBKDF1 from PKCS#5/rfc2898 but modified, and is implemented by the API function EVP_BytesToKey whose man page should be available in any Unix system with OpenSSL installed and on the website. A bit confusingly, and inconsistent with more recent best practice, this 'key derivation' actually derives both the key and IV if applicable. To lay out the relationship exactly:

    • PBKDF1 uses a choice of hash from MD2 MD5 SHA1, and an iteration count, and (required) salt. It initially forms password || salt and hashes it, then hashes the first hash, then hashes the second hash, iterating to a total of count times. PKCS5 PBES1 uses PBKDF1 to produce both key and IV -- but only for RC2 and (single) DES, neither of which is acceptable today.

    • EVP_BytesToKey uses any hash supported by EVP, which may depend on build options but currently defaults to MD4 MD5 MDC2 RIPEMD RIPEMD160 SHA1 and the original SHA2 family (but not the later '512 slash' additions), plus iteration count and optional salt. It does one block like PBKDF1 as H^count (password||salt) with the salt omitted if not used, but if the required 'key material' is more than one block, it then does additional blocks as H^count (prevblock||password||salt) ditto. Whether the EVP_BytesToKey result is used for IV depends on the caller.

    • enc uses EVP_BytesToKey with salt by default (but see options above); (edit) hash MD5 by default through 1.0.2 or SHA256 by default in 1.1.0 released 2016-08 but you can specify otherwise with the option -md which was only documented from 1.0.0 but actually available before; and iteration count ONE. Thus the key and IV if applicable used by enc before 1.1.0 defaults to

      key[||IV] = b0=md5(t=pw[||salt]) || b1=md5(b0||t) || b2=md5(b1||t) ...
      

    This is a poor PBKDF. Because all the available hashes (including default MD5 or SHA256) are fast and this PBKDF 'iterates' only once, an attacker can quickly try large numbers of possible passwords to find yours, unless your password contains enough entropy without any significant 'strengthening' -- and most people don't choose, and even if given can't remember, such strong passwords. But we are stuck with it for backward compatibility.

Thus for examples:

openssl rc4-128 [-e|-d] -k sekrit -nosalt 
# uses RC4 with 128-bit key (RC4 is a stream cipher and uses no IV)
# derived using no salt, MD5 (unless 1.1.0), and count 1 from 'sekrit'

openssl aes-256-cbc [-e|-d] -k sekrit -md sha1
# uses AES in CBC mode with 256-bit key and 128-bit IV 
# (both) derived using random salt, SHA1 and count 1 from 'sekrit'
# and adds the Salted__ header with the salt to the ciphertext

And finally, if you specify -base64 or the abbreviation -a, the ciphertext (including header if any) is encoded to base64 on encryption, and decoded from base64 on decryption; this is fairly commonly needed because some applications, systems, or protocols cannot handle the arbitrary (quasi-random) binary bytes needed for ciphertext. (edit) Similarly if you specify -z plaintext is compressed before encryption or decompressed after decryption, but only if OpenSSL was built with compression (zlib), and after the CRIME and BREACH attacks some builders or packagers disable compression entirely. These options are independent of the cipher and key/PBKDF, and in fact you can use them alone to only base64 encode/decode or zlib compress/decompress without doing encryption or decryption at all.

Base64 is used with slight variations in many standards but OpenSSL mostly follows the original PEM in 4.3.2.4 (no internal hyperlink because 1993 was before WWW became popular). Two caveats: base64 decoding silently ignores part and sometimes all of a line with a non-base64 character; this is useful when OpenSSL uses it internally to read PEM-format files, but may not be so in other applications. And before 1.1.0, it silently ignores lines longer than the 76-character limit specified in MIME unless you add -A (uppercase). zlib compression is related to, but not quite the same as, gzip.

Inter-version compatibility: (added) using password-based encryption and decryption depends critically on using the same parameters and especially the same hash in the PBKDF for both operations. As noted above, the default hash used by the enc command changed in 1.1.0 to SHA256 versus MD5 in lower versions. Thus data encrypted using the default hash in lower versions won't decrypt with the default in 1.1.0 or vice versa.

  • if data was encrypted with the default (MD5) on an older version, decrypt in 1.1.0 (and presumably higher when released) with -md md5

  • if data was encrypted with the default (SHA256) on 1.1.0 (or presumably higher), decrypt in an older version (at least back to 0.9.8, possibly before) with -md sha256

  • to proactively prevent a problem, specify -md consistently for both encryption and decryption even when it's redundant

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.