How does TLS 1.3 provide authentication without using the public key to verify that the server has a private key?

Question

In the TLS 1.2 handshake, after checking the certificate, the public key from the certificate was used to encrypt the data to create a symmetric encryption key, hence the authentication took place on the factor of knowing the private key, as it is needed to decrypt that data.

In TLS 1.3, the public key cannot be used to encrypt data needed to create a symmetric encryption key because this data is included in the first message sent by the client and the client cannot know the public key until it receives the certificate that the server sends in response to the first message.

Since the public key is not used to encrypt data needed to generate a symmetric encryption key or any other data the private key knowledge factor disappears.

How does TLS 1.3 provide authentication and what is the point of using PKI and Public Key Certificate if the public key is not used to verify that the party has a private key?

Answer 1

... the public key from the certificate was used to encrypt the data to create a symmetric encryption key, hence the authentication took place on the factor of knowing the private key, as it is needed to decrypt that data.

This was not the authentication. What you describe is the RSA key exchange, which was removed in TLS 1.3 and is even considered obsolete with TLS 1.2.

The proof that the server knows the private key is instead done with the CertificateVerify message, which is (slightly simplified) a signature (using the private key) over the messages previously send in the handshake. This means it also includes client side random data and thus can not be simply replayed within another handshake.

Answer 2

The key exchange in TLS 1.3 is mainly based on the SIGMA protocol, implementing a "SIGn-then-MAc" variant of SIGMA. I will only describe high-level ideas for the full handshake, other considerations are needed for other handshake like the PSK mode. For an extensive security analysis, I recommend "A cryptographic analysis of TLS 1.3 handshake protocol" by Dowling, Fischlin, Günther and Stebila.

How does TLS 1.3 provide authentication

Zooming out of TLS 1.3, SIGMA builds on an unauthenticated DH and authentication in SIGMA is guaranteed by signature over the communication transcript of the key exchange and a MAC over the identity of the authenticating party. The basic SIGMA flow with mutual authentication between a Client C and Server S is as follows:

• C -> S: $g^x, \text{identity(C)}$.
• S -> C: $g^y, \mathrm{sig}(sk_S, g^x, g^y), MAC(identity(S))$
• C -> S: $\mathrm{sig}(sk_C, g^x, g^y)$

SIGMA is shown (here for example) to provide strong authentication guarantees, key secrecy and forward secrecy even assuming a strong attacker; i.e.: an attacker that fully control the network, that can reveal established session keys, compromise the long-term signing keys of some entities. As a consequence, TLS 1.3 inherits those guarantees, though it makes a few changes like deriving the keys with a more complex key schedule that incorporate the entire transcript in the key schedule.

what is the point of using PKI and Public Key Certificate if the public key is not used to verify that the party has a private key?

Firstly, as seen in the SIGMA flow. The public keys are used to provide signature during authentication. Additionally, a PKI serves as a trusted way to bind identities to public key. This is necessary since a malicious party can impersonate another without any ground truth regarding identities and public keys.

Note that, technically, SIGMA doesn't take maliciously generated public keys into it's threat model. But this is another issue.