Honestly I'm really confused about padding in RSA scheme. I don't study computer science, and I'm a beginner learning cryptography (self-taught). Can anyone give me suggestions?
Direct use of textbook RSA for encryption is very insecure. For example, a guess of the plaintext can be checked: if what's encrypted is a name on the class roll, enciphering all names on the class roll until getting the ciphertext breaks confidentiality. There are several other attacks.
That motivates RSA-OAEP. It turns textbook RSA, a hash, and a source of random bits, into a demonstrably secure encryption scheme. Encryption of message $M$ first transforms $M$ into padded message $X$ by mixing it with random bits and applying reversible transformations with the hash; then enciphers $X$ into $X^e\bmod N$, that is per textbook RSA encryption. Decryption starts by textbook RSA decryption, yielding $X$; then the padding is undone, yielding $M$ or an error indication.
I essence, the distribution of $X$ is random enough that textbook RSA becomes secure. A proof of that can be made for the strongest definition of security of a cipher, IND-CCA2, and under some hypothesis:
- the RSA problem of finding a random $X$ given $X^e\bmod N$ is hard;
- the hash is undistinguishable from a public function selected at random;
- the source of random bits is indistinguishable from one producing uniformly random and independent bits;
- a decryption device given a cryptogram only outputs the correct plaintext or an error indication (it does not leak any other information).
A (highly technical) reference proof is in Eiichiro Fujisaki, Tatsuaki Okamoto, David Pointcheval, and Jacques Stern's RSA-OAEP Is Secure under the RSA Assumption, in Journal of Cryptology, 2004. A more gentle text is David Pointcheval's How to Encrypt Properly with RSA?, originally in Cryptobytes 5n1, 2002.
The practical form of RSA-OAEP is named RSAES-OAEP and defined in PKCS#1 v2.2.