how hashing algorithm is shared in public key

Question

I am reading about public key working at following linkenter link description here

Here picture is provided

My question is how hashing algorithm is shared between sender and receiver?

Answer 1

First: This picture is wrong. Signature is not encryption with the private key. Encryption with the private key is a category error; it doesn't make sense. This suggests to me that whatever source you are reading is likely to be full of confusing misinformation.

That said: The signer and verifier have to agree in advance what signature scheme to use, which entails a choice of hash function.

In brain-damaged protocols, the signature or metadata around it specifies which signature scheme to use (including the choice of hash function), and the verifier will happily use it. This means that a forger can pick any weak signature scheme that the verifier supports—including completely nonsensical ones, like none, or the secret-key HMAC-SHA256 when the signer meant to use the public-key RSASSA-PSS—and persuade the verifier to use it, against the wishes of the signer.

In your system, in order to avoid leaving a trail of devastation and screaming cryptographers in your wake like JWT did, you should have the verifier store alongside the public key (not the potentially forged document!) whatever metadata you need to specify a particular signature scheme for that public key. For example, in x.509 certificates, alongside the public key there is a field specifying what cryptosystem the public key is meant for. Then there is no need for the signature to control which signature scheme the verifier uses: the verifier already knows for each public key which one to use.

As a simplification, you can just use a single signature scheme in your application, Ed25519, and never reuse keys in your application for any other purpose. Then there's no need for any metadata. If you ever need to replace the signature scheme, just replace the keys too.