Is this the main way of encrypting with PGP, ssh, ssl (https), DKIM, ...?
I wonder if Diffie-Hellman is the clue to understand how encryption with public and private keys work and why it works
As the name says Diffie-Hellman key exchange is a key exchange protocol, i.e., a protocol where two parties agree on a common secret without having exchanged any secret prior to that, in an interactive way, i.e., both parties are online. The so agreed common secret can then be used for instance to derive keys for symmetric encryption and/or message authentication (as it is for instance done in ssl/tls, but this protocol can also be used with e.g., RSA encryption). Consequently, key exchange protocols can be used to agree on a key whenever the protocol involves interaction and is applicable in context of ssl/tls, ssh, etc.
In PGP, e.g., used for email encryption, you can use for instance the ElGamal public key encryption scheme, which is related to Diffie-Hellman but non-interactive. You can look, e.g., on this question for the difference between Diffie-Hellman and ElGamal public key encryption. Note that PGP and other related applications do not assume that the other party is online and so they can not use interactive key exchange protocols. But all such applications use public key encryption only as a means to transport some symmetric key (hybrid encryption) to the receiver along with some data encrypted under this symmetric key.
An interactive key exchange protocol is no public key encryption scheme (there is no pivate key that can be used to decrypt anything, it results simply in a shared key that both parties hold).
There are various other public key encryption schemes that are totally unrelated to the idea behind Diffie-Hellman key exchange, such as for instance RSA encryption (which is based on a trapdoor one-way function - whereas there is no trapdoor in the DH setting).
So understanding Diffie-Hellman key exchange may help you to understand how ElGamal (one particular public key encryption scheme) works, but may not be enough to understand the idea behind public key encryption in general.