Untrusted certificate or expired certificate?

If you go strictly by RFC 5280 ordering, "untrusted certificate" should be returned.

More specifically section 6.1.3 starts with

   The basic path processing actions to be performed for certificate i
   (for all i in [1..n]) are listed below.

      (a)  Verify the basic certificate information.  The certificate
           MUST satisfy each of the following:

         (1)  The signature on the certificate can be verified using
              working_public_key_algorithm, the working_public_key, and
              the working_public_key_parameters.

         (2)  The certificate validity period includes the current time.

As RFC 5280 goes the from signer to signee, the current working_public_key is the ones of all trusted CAs (section 6.1.2 (h)). As the CA in question is not trusted, no public key will yield a successful validation. As this is checked before the expiration, it should yield the error.

Note that this also makes sense from a cryptographic standpoint, as you always want to make sure that your data is authentic before parsing it to avoid malicious data (like maliciously crafted certificates) to break and exploit your parser.

Untrusted certificate or expired certificate?

If you go strictly by RFC 5280 ordering, "untrusted certificate" should be returned.

More specifically section 6.1.3 starts with

   The basic path processing actions to be performed for certificate i
   (for all i in [1..n]) are listed below.

      (a)  Verify the basic certificate information.  The certificate
           MUST satisfy each of the following:

         (1)  The signature on the certificate can be verified using
              working_public_key_algorithm, the working_public_key, and
              the working_public_key_parameters.

         (2)  The certificate validity period includes the current time.

As RFC 5280 goes the from signer to signee, the current working_public_key is the ones of all trusted CAs (section 6.1.2 (h)). As the CA in question is not trusted, no public key will yield a successful validation. As this is checked before the expiration, it should yield the error.

Note that this also makes sense from a cryptographic standpoint, as you always want to make sure that your data is authentic before parsing it to avoid malicious data (like maliciously crafted certificates) to break and exploit your parser.