What exactly is Let's encrypts chain.pem file?

Question

While trying to understand the use or meaning of the fullchain.pem file created by let's encrypt I stumbled upon this post in which fullchain.pem is explained as:

fullchain.pem is a concatenation of cert.pem and chain.pem in one file. In most servers you’ll specify this file as the certificate, so the entire chain will be send at once. Some clients require you to specify the above two files separate. In that case you won’t need

While chain.pem on the other hand is described as

chain.pem contains the intermediate certificate, the certificate from Let’s Encrypt containing the public key which is “coupled” to the private key which signed your certificate (the one above). This intermediate certificate is required for clients to verify your certificate;

But I have troubles really understanding what this means. By this I mean what exactly is intermediate certificate and where does it come from.

I understand the concept of CA's and the fact that CA issue certificates. I understand that the certificate issued by the CA which is put to use is the end user certificate.

I also understand the chain of trust, where there are intermediate CA between the end user certificate and the root CA.

What I can't understand is how let's encrypt can know all the possible intermediate CA between the root CA and the end certificate.

I mean it is possible to know all the CA, that the root CA itself signed. But what if those other CA's created other CA's that ends up issuing the end user certificate, how can this ever be in chain.pem?

Which then begs the question, how can those end user certificate's be ever verified, since it seems the full chain of trust, the full intermediate CA is not known in chain.pem?

Anyone kind enough to help me understand the setup? Thanks!

Answer 1

When requesting a certificate one sends a Certificate Signing Request, containing the information to be signed by a CA, to that CA. The CA responds with the resulting certificate and the chain of certificates back to a root. That chain is the CA's certificate, any intermediate CA certificate(s) that signed the CA, (possibly multiple levels), and the root certificate that signed the top-level intermediate CA certificate (if that's not the same as the CA's certificate in question). In Let's Encrypt's case, there's only the CA certificate and the Root in this chain.

The CA knows everything in the chain because they in turn had requested their certificate from an intermediate CA or root, and they know who they requested it from. So they know the certificate of the intermediate or root, and can include it. Likewise back up the chain.

Root CAs have self-signed certificates. There's no other entity in their certificate chain. These certificates get imported as trusted root certificates by browser vendors (and other entities dealing with the TLS CA system), allowing verification of certificate chains starting at these roots.

Since every time anyone (including a CA) requests a certificate they get back the entire chain it's trivial for a CA to send the chain along with the resulting certificate from a request. Also, certificates are public and all major CAs publish the certificates they issue in "certificate transparency logs", so even without the chain one could (in theory) look up the certificates for each signing entity.

Answer 2

What I can't understand is how let's encrypt can know all the possible intermediate CA between the root CA and the end certificate.

Only the intermedate certificate that constitutes the chain from the root certificate to the server certificate is needed. How very fortunate.

Answer 3

Intro:

I bumped into this issue with Let's Encrypt and had the same questions as the OP, so here's the technical detail I learned as to the mechanics of how openssl can validate a cert.

Validating:

The "Intermediate" cert will not contain the CA cert, but yet we require this to trace the Chain of Trust back to an origin. Hence the term "Intermediate"- it's not inclusive of all certs used to build the complete Chain of Trust.

When you validate a cert with:

openssl verify -untrusted chain.pem cert.pem

openssl will parse the "Trust Store" on the server (or container) until it finds a match. So although we haven't expressly specified a "Trust Anchor" with the -trusted or -CAfile switches, openssl verify can nonetheless find the related cert.

On a linux system when you install the package ca-certificates this will populate the directory with CA Certs that openssl will parse.

To find the path of "Trust Store" that openssl is using to trace the Chain of Trust:

openssl version -d

Bonus:

To actually see what certs are included in a bundle of certs using Let's Encrypt as an example, just use the following commands replacing example.com with the domain you used to register your Let's Encrypt cert:

while openssl x509 -noout -text; do :; done < /etc/letsencrypt/live/example.com/cert.pem

while openssl x509 -noout -text; do :; done < /etc/letsencrypt/live/example.com/chain.pem 

while openssl x509 -noout -text; do :; done < /etc/letsencrypt/live/example.com/fullchain.pem