The certificate isn't a secret. If all that's needed to impersonate a server was the certificate, everybody who ever connected to that server (which doesn't reject connections!) would be able to impersonate him.
What instead happens is the following:
The server sends a piece of data (the certificate) which binds a public key to a domain and potentially a company. This binding is done by a trusted CA and enforced with a signature which the client can verify.
In the next step one of two things happen:
- The client encrypts the so-called pre-master secret with the public key from the certificate (which he knows belongs to the legitimate owner of the domain) OR
- The server signs another (fresh) (DH) public key which the client can verify was known to the legitimate owner of the domain. Now the client responds with his (fresh) (DH) public key.
Now there's a piece of information that only the legitimate owner of the private key associated to the certificate can know: Either the pre-master secret OR the private (DH) key. No matter which information is shared, now a "master-secret" is generated which cannot be pre-calculated because it includes randomness from both the server and the client. To finally verify both parties share the same master-secret (which is only possible if the server actually knew the private key from the certificate) they both exchange "secret-key-signatures" over all previous handshake messages and thus assure each other that they know the secret key. The identity of the server is thus confirmed (and not earlier than this).