What is the encoding protocol for passphrase protected RSA private keys?

Question

What is the protocol used to encode an RSA key protected with a passphrase?

If I generate an RSA key with

ssh-keygen -t rsa -b 4096

and do:

head ~/.ssh/id_rsa

I get:

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,83F1F6710FBD0610A70A04F5ABDE4114

<base64 data>

What encoding protocol does this use?

For RSA private keys not protected by a passphrase, they are encoded in Base64 ASN.1 and the structure is documented here: https://www.rfc-editor.org/rfc/rfc8017#appendix-A.1.2

Answer 1

This file format, which is used by OpenSSH below 7.8 by default for RSA, is actually defined and implemented by OpenSSL, which documents it fairly tersely on the man page for a group of library routines including PEM_write_RSAPrivateKey near the bottom at the heading "PEM ENCRYPTION FORMAT". Quoting for convenience and because of many Stack-ers' linkophobia:

The line beginning with Proc-Type contains the version and the protection on the encapsulated data. The line beginning DEK-Info contains two comma separated values: the encryption algorithm name as used by EVP_get_cipherbyname() and an initialization vector used by the cipher encoded as a set of hexadecimal digits. After those two lines is the base64-encoded encrypted data.

The encryption key is derived using EVP_BytesToKey(). The cipher's initialization vector is passed to EVP_BytesToKey() as the salt parameter. Internally, PKCS5_SALT_LEN bytes of the salt are used (regardless of the size of the initialization vector). The user's password is passed to EVP_BytesToKey() using the data and datal parameters. Finally, the library uses an iteration count of 1 for EVP_BytesToKey().

The key derived by EVP_BytesToKey() along with the original initialization vector is then used to decrypt the encrypted data. The iv produced by EVP_BytesToKey() is not utilized or needed, and NULL should be passed to the function.

EVP_BytesToKey itself is documented on its own man page and covered in numerous Stack questions or answers including one of my longer ones (which is actually about commandline enc but the EVP_BytesToKey part applies additionally to 'legacy-PEM' encryption).

The decrypted plaintext is the same as the unencrypted form, PKCS1 (rfc8017 et pred).

The ursine answer to this Q over in security also covers this and related formats in some detail.

For clarity note there are many other file formats for RSA keys, even within OpenSSL and OpenSSH much less other software; this is a scheme for some passphrase-protected RSA private keys but not the scheme for all such keys.