If someone gives me their PGP key, can I use it with GPG, and vice versa, all the time (100% interchangeable)? Or are there times when they are not compatible, when only PGP can be used with a PGP key, and only GPG used with a GPG key?
"If PGP and GPG both follow the OpenPGP standard, are they 100% compatible in all use cases?"
No, they are not 100% compatible in all use cases, because — depending on the PGP version — there are known interoperability problems.
The GNUPG FAQ answers this question quite well:
Is GnuPG compatible with PGP?
In general, yes. GnuPG and newer PGP releases should be implementing the OpenPGP standard. But there are some interoperability problems. See question How can I encrypt a message with GnuPG so that PGP is able to decrypt it? for details.
For your convenience, I'll also quote that linked Q&A:
How can I encrypt a message with GnuPG so that PGP is able to decrypt it?
It depends on the PGP version.
You can't do that because PGP 2.x normally uses IDEA which is not supported by GnuPG as it is patented (see How do I include support for RSA and IDEA?), but if you have a modified version of PGP you can try this:
$ gpg --rfc1991 --cipher-algo 3des ...
Please don't pipe the data to encrypt to gpg but provide it using a filename; otherwise, PGP 2 will not be able to handle it.
As for conventional encryption, you can't do this for PGP 2.
PGP 5.x and higher
You need to provide two additional options:
--compress-algo 1 --cipher-algo cast5
You may also use "3des" instead of "cast5", and "blowfish" does not work with all versions of PGP 5. You may also want to put:
into your ~/.gnupg/options file — this does not affect normal GnuPG operation.
This applies to conventional encryption as well.
I think that pretty much answers your question.
From my recent experience, it's not always compatible. Using the latest version of GnuPG and Symantec PGP, I was able to confirm at least the fact that a 16384-bit RSA key pair (with 4096-bit subkeys) generated in GnuPG will not be usable in Symantec PGP. All of the cipher, hash algorithm, and compression preferences will be displayed as none, the expiry date will be Never, and you cannot even encrypt with the key. The cipher, hash algorithm, and compression preferences cannot be changed either leaving you with a key that is completely unusable. Of course, 16384-bit keys are not guaranteed to work in standard versions of GnuPG either. I've yet to explore the full extent of the compatibility between the two pieces of software, but I suspect the only real way to ensure compatibility is to stick with the lowest common denominator options in both programs.