This is a very good question.
Now, in principle, a public-key certificate works like any other kind of certificate, including the ones in the real world.
- If some authority A asserts some property p(X) for some X
- If A publishes the assertion in a kind-of unforgeable way
- If some entity Y trusts A
then Y has virtually no choice but to believe that p(X) is true.
For public-keys in a X.509-style PKI that means, A is a CA, p(X) is a claim along the lines of "the Key K herein belongs to entity X" and the unforgeable way is via secure digital signatures with adequate key-lengths (for simplicity, I assume that all relevant algorithms are implemented flawlessly...).
Moreover, the following conditions are crucial:
- Y needs to have a genuine copy of CA's public key.
- CA must be trustworthy, which means, the know what they do, they are not corrupt or being corrupted themselves.
- All devices involved are trustworthy (i.e. not infested by trojans, backdoors and such)
Each of those are quite tricky and hard to achieve - if at all.
Acquiring a public key in a trustworthy way
I think this is the issue that gave rise to the question. Since anyone can create arbitrary key pairs and from this, create arbitrary certificates, the trick is to distinguish between the wheat and the chaff, i.e. to tell genuine and fake root certificates apart. This can be done by secure channels, but for that you need some kind of trustworthy "seed". For instance, if you consider how the Web Of Trust works, particular the legendary PGP key signing parties, all involved attendees meet in person and mutually check photo IDs (another brand of certificate...). However, that does not scale.
On the other hand, if you consider how root certificates are distributed in web browsers and operating systems, you find that you depend on the software vendors. Now, once upon a time there was the vision of a global PKI with one ultimately trusted root, where a copy of the public key or digital fingerprints thereof would be virtually everywhere posted (e.g. in printed publications). Of course, there was/is no agreement conceivable on who that could be, and thus this position remains vacant. Well, not quite, the software vendors silently filled that vacancy, and this seems to be accepted on the ground that there is nothing better in sight for the moment, and somehow, this seems to work (but see below).
Closed organisations like enterprises and government bodies are a special class of cases - whenever you enter the organization, you get your PC, your accounts, and alongside you get your key pair and the public key of the organization root CA (more precisely, this is typically stored in the Windows Active Directory, where the AD architecture with its protocols like Kerberos provide the secure channel).
CA trustworthyness
Running a root CA properly is not easy, especially since they are a very attractive attack target. You need detailed procedures, background checks of the staff, regular training, suitable rooms, etc. etc. etc. And yes, there are audits by the browser forum and possibly other bodies, but the famous glitches of Commodo and DigiNotar show how hard it is to run a CA properly.
Device trustworthyness
This is another property that is hard to achieve. General-purpose hardware with general-purpose OSes allow you to install any software that you want - and coincidentally, any software that you don't want, including malware. The average textbook describes an encryption process (say) as Alice encrypts message M with her key K or something like that. But who is Alice? If that refers to a person, then the statement is to be read as Alice instructs her computer to ... . Given the Trojan malware that we saw in the past (particularly Trojans for "alternative" online banking) it is easy to conceive that the computer does something quite different than intended or expected while pretending otherwise.
Now what?
To be honest, nobody knows. Bruce Schneier and others have written a lot about that. But as far as I know (I am not closely watching, though) the whole construction was never thoroughly contested in the courts. Until then it hold what I said earlier: The existing architecture somehow works, despite the many leaps of faith you have to make, and alternatives that are widely accepted are not in sight.