How do cryptographic systems handle such situations? … What actually is done in real life?
In real life, cryptography handles situations like the one you describe by using “authentication”. Authentication links an action, a message or a situation to an identity. In the example you describe, this will practically boil down to a validation of something an entity knows… like a password, a key, or another kind of secret.
Simplified, you could think of authentication as something which enables you to validate if an entity is actually who they claim to be, or allowed to do something, by showing they know a secret that only they could know. (Note that “authentication” does not always mean an entity proves its individual identity; it can also be used to prove an entity is part of a group of entities knowing a certain secret.)
For better understanding of what “authentication” is, it may be helpful to know there are also other forms (better: factors) of “authentication”. Chances are, you’ve already have met such “authentication” in real life yourself, since “authentication” can be proven by something an entity has – like a passport, a smartcard, a token device, or something similar.
Besides that, “authentication” can be proven by an entity by something an entity is. For example: humans have specific, individual features. These can also be used for “authentication”… which makes up one of the areas biometrics covers. If you‘ve ever had your fingerprint or iris scanned, or if you ever saw that in a movie… well, that’s what we call “authentication” in the realms of cryptography.
As said, in your example Bob and Alice will use a shared secret to enable mutual authentication. Not knowing the secret, Mallory will not be able to successfully authenticate her messages as Bob will detect the missing or wrong authentication factor (the secret) and reject and/or ignore Mallory’s messages/attacks accordingly.
Of course, authentication alone won’t make a protocol secure. Besides authentication things like secure connections (using encryption, etc.) are needed to wrap a secure shield around things. But authentication is an important part of cryptography, as it allows (for example) to learn if the entity you are communicating with is indeed the entity you expect to be communicating with… and not some kind of adversary who tries to inject or even replace messages which you might think are secure. They may be secure due to encryption, but without authentication that’s not worth all too much. Yet, the combination of both make up a pretty solid way of exchanging information between two entities.