If someone gives me their PGP key, can I use it with GPG, and vice versa, all the time (100% interchangeable)? Or are there times when they are not compatible, when only PGP can be used with a PGP key, and only GPG used with a GPG key?
No, they are not 100% compatible in all use cases, because — depending on the PGP version — there are known interoperability problems.
The GNUPG FAQ answers this question quite well:
For your convenience, I'll also quote that linked Q&A:
I think that pretty much answers your question.
From my recent experience, it's not always compatible. Using the latest version of GnuPG and Symantec PGP, I was able to confirm at least the fact that a 16384-bit RSA key pair (with 4096-bit subkeys) generated in GnuPG will not be usable in Symantec PGP. All of the cipher, hash algorithm, and compression preferences will be displayed as none, the expiry date will be Never, and you cannot even encrypt with the key. The cipher, hash algorithm, and compression preferences cannot be changed either leaving you with a key that is completely unusable. Of course, 16384-bit keys are not guaranteed to work in standard versions of GnuPG either. I've yet to explore the full extent of the compatibility between the two pieces of software, but I suspect the only real way to ensure compatibility is to stick with the lowest common denominator options in both programs.