- What is RSA used for during key exchange?
RSA can be used either (a) for signing the key agreement transcript with some other mechanism of key agreement (ciphersuites
TLS_*DH*_RSA_WITH_*), or (b) for the client to encrypt a premaster secret key it sends to the server as an authenticated key agreement (ciphersuites
The premaster secret determines the key under which records exchanged by the peers in the session are symmetrically encrypted and authenticated, e.g. with AES-GCM.
In both cases, the server does a private key operation to prove its identity: in (a), it signs the key agreement transcript with the server's long-term public key, producing a signature that the client (and anyone else) can verify; in (b), it decrypts the client's premaster secret key, proving it is able to decrypt messages encrypted to the server's long-term public key.
Note that in TLS 1.3, RSA key agreement is going away.
- What is Diffie-Hellman used for?
Diffie–Hellman, over finite fields or over elliptic curves, can be used for ephemeral or static key agreement.
In the ephemeral case, both parties generate and exchange a fresh key pair and use it to agree on a premaster secret key, which they then authenticate by another means, such as an RSA or ECDSA signature. (Note that in principle you could use DH for key agreement and authentication, but nobody does this in TLS.)
In the static case, at least one of the parties, typically the server, has a long-term DH public key, which the other party, typically the client, combines with a DH private key to derive a premaster secret. The server proves its identity by deriving the same premaster secret from its private key and the client's (ephemeral or static) public key. (Static DH isn't used much.)
- Since RSA already provides authenticity and confidentiality, why is Diffie-Hellman used during the exchange?
Diffie–Hellman provides a fast key agreement procedure, with a small number of rounds trips, that supports fast key erasure: as soon as the session is done, all copies of the DH private keys, the derived premaster secret, the derived master secret, etc., can be erased by the peers. Fast key erasure means compromising the long-term identity keys does not enable retroactive decryption of the session.
In contrast, with RSA key agreement, future compromise of the server's long-term identity key trivially enables an adversary to retroactively decrypt past sessions, because the session keys were all encrypted to the server's long-term identity key.
- I've heard DH exchanges aren't secure. Is that something to be worried about in an RSA with Diffie-Hellman exchange?
There are lots of security issues one might be concerned about. Maybe your friend is concerned about historical mistakes in supported ciphersuites; maybe your friend is concerned about timing side channels in finite-field arithmetic over general primes; maybe your friend is concerned about dynamic DH group choices and insufficient validation; maybe your friend is concerned about choices of elliptic curves. But I can't read your friend's mind: you'll have to be more specific.
- If DH is used for forward secrecy. What's wrong with just generating new RSA key pairs and exchanging keys with that?
You could generate a fresh RSA identity key for every session, ask your certification authority to sign a certificate for that key, and then use that for RSA key agreement.
This would be the most stupidly expensive TLS stack ever, because generating RSA keys is extraordinarily expensive compared to generating DH keys (and takes an unpredictable amount of time to boot), sensible certification authorities store their keys in hardware security modules that you don't want to have attached to your TLS front ends, and sensible certification authorities have elaborate audit trails around certificates including Certificate Transparency logs that you can explore at crt.sh.
You could devise an RSA-based key agreement protocol with ephemeral RSA keys and without involving your CA, but RSA keys are still extraordinarily expensive to generate, and it would take additional round-trips between the peers, and you would still have to find some way to authenticate them.