Is there some standard protocol for mutual authentication of two parties when there is possibility to use public key cryptography? The problem is that two parties can use public key cryptography but public keys are not embedded within any kind of certificates. Is there a way to avoid MITM attacks? Simple use of public keys for authentication does not solve the problem of MITM attack (I would have to be sure about public key authenticity first). If I understand correctly FIPS 196 requires usage of authenticated keys so this is no solution. Is it possible? If there is no previous communication and no certificates then how would it be possible for Alice and Bob to recognize that there is Eve trying to do something nasty? And if there would be possibility for Alice to verify the fingerprint of Bob's public key that would be possible? Is there some standard protocol for this?
Without an out-of-band channel, no.
If all Alice has only a public key , she can't tell the difference between Bob's key and Mallory's. Hence Mallory can mount a man in the middle attack.
To prevent this you either need a certificate or a a trusted out of band channel though which you conform the key. The channel could be something that is harder to man in the middle ( e.g. the phone network) which you use to exchange key fingerprints, it could be an in person meeting(some people of pgp fingerprints on their business cards), or it could be that you remember the key from the last time you connected ( which only prevents subsequent man in the middle attacks)
A somewhat novel application of this is used the ZRTP key agreement protocol used in VOIP. ZRTP generates a short authentication string(SAS) based on the user's keys. Bob can read that SAS to Alice and assuming Mallory can't forge Bob's voice in realtime, the channel is secure and the key is authenticated.
There are some efforts, particularly in distributed environments such as P2P networks, to come up with solutions that allow identity verification with a certain degree of certainty. One such effort is the one presented on S/Kademlia (a DHT) which includes a puzzle in the generation of the node id for each peer https://doc.tm.uka.de/SKademlia_2007.pdf