TLDR: No encryption with a private key. To sign a message in Java, Signature
is appropriate.
Encryption is not performed with the help of a private key. Hence in the first option of the question
calculated the hash and encrypted it with the private key of sender
and in the second option
signature is created by an hash and same is encrypted, it is done by private key of sender
both are terminology errors. In either case, what's described is not encryption. At least the intent is signature production. I see no fundamental difference between these two actions as detailed in the question, when assuming that in the second "same" refers to the hash (and disregarding the question's title and first paragraph). If on the other hand we read "same" as "similarly to the first option", then there often is a security difference. That's the reading in this other answer.
Encrypting a hash is possible, but is never done with a private key used as such. If something on that tune is attempted, that's not encryption, that's signature, or meaningless, or impossible. In the case of RSA (but not many other asymmetric cryptosystems), it is possible and meaningful to compute $(H(M))^d\bmod N$ where $(N,d)$ is the private key, and the outcome is a signature of $M$, of the appendix kind. It is more or less secure¹. Proper RSA signature schemes differ from that by the choice of $H$, which becomes a signature padding, and may incorporate a random input and/or recoverable information conveying some or all of the message.
Encrypting a hash can only be meaningfully performed using (1a) symmetric encryption with a secret key, or (1b) asymmetric encryption with a public key. There's a major difference with (2) signing with a private key:
- Anyone with the trusted public key can verify the signature of (2) and that can prove integrity and origin of the signed message, but not in the other cases.
- The outcome of (1a) can be verified only with the secret key, and can at best prove to a party holding the private key that the message was approved by another party holding the secret key. Thought with many encryption schemes in common use (including stream ciphers), that goal is missed and forgery is easy.
- The outcome of (1b) can be decrypted by the holder of the private key matching the public key, but its virtue is different: proving that whoever produced that cryptogram knew the hash of the message, or/and conveying that hash to the holder of the private key, without revealing the message or its hash to another party (this is not a commonly pursued goal, and it has no name).
¹ Security depends on the width of $H$ and attack model. In particular, a collision-resistant hash such as SHA-224 or even SHA-256 is not suitable in an attack model where the adversary can obtain the signature of chosen messages: forgeries may be possible, see this.