Assuming I am using some derivation function to generate encryption key from the string to encrypt a file. How that string is later verified during decryption?
If user enters wrong password/string, why system / app doesn't decrypt entire file resulting in incorrect clear text? What mechanism is behind the scene?
It depends on the protocol, there is not one way of handling this.
If a symmetric cipher does not provide integrity protection or message authenticity (through a MAC calculation) then it may well be that the decrypted plaintext is not correct.
Usually we try and use an authenticated cipher like GCM to avoid such problems. In that case the verification of the authentication tag fails and the file is not decrypted.
If the ciphertext is large it may be a good idea to create a separate MAC over some known information and include it with the ciphertext. That way the validity of the key can be determined without having to decrypt the entire message. It may also help detecting that e.g. the decrypted file isn't fully present in it's original form. It seems wasteful to perform the full verification of the ciphertext in case a wrong password is provided, and it makes no sense to have more security for large files as for small files in case an adversary tries to find the password.
Note that above trick is basically an enhanced version of a Key Check Value, and the usefulness of KCV is debatable. There is no good way of distinguishing between a bad password and an altered ciphertext. Even if you include an additional MAC with the ciphertext, then that MAC value may be altered as well.
Obviously providing a signature over the ciphertext may also work. You must make sure that the public key of the sender is trusted by the receiver, otherwise signature verification is useless. Do note however that other trusted parties - if any - could remove the signature and sign the file themselves.
