The standard convention in asymmetric cryptography (thus RSA), is that key generation generates a key pair comprising a public key and a private key, and unless otherwise stated
- the public key is public, that is known to all;
- the private key is secret, only known by who/what generated the key pair.
Asymmetric encryption (thus RSA encryption) aims at transforming plaintext into ciphertext using only public information, such that getting back at the plaintext is possible only by authorized parties holding a secret. It follows that
- encryption is with the public key;
- decryption is with the private key.
Signature (thus RSA signature) aims at transforming plaintext into a signed version (e.g. by appending a signature) such that anyone can verify the message's integrity. It follows that
- signing is with the private key;
- verifying is with the public key.
That said, in RSA (and very few other asymmetric cryptography schemes), if the public key and private key are expressed as pairs of integers $(n,e)$ and $(n,d)$, it's possible to interchange the values of the keys and still perform a computation. If so,
- If we perform this exchange before making the public key public
- If (as customary) the original public exponent ($d$ after the exchange) is small, or/and chosen in a non-secret way, then all security is lost, because the new private exponent can be found (in particular this attack is relevant).
- If the original public exponent is large enough, random, and kept secret, then the key pair can be secure, but the exchange serves no purpose.
- If we perform this exchange after making the public key public (thus making the new private key public)
- Textbook RSA encryption of a suitably small plaintext becomes textbook RSA signature (or production of the signature if we define textbook RSA signature as producing an appendix to the message). That's the case when selecting "RSA Key Type: Private Key" and "Decryption Algorithm: RSA" in the question's linked tool, then "Encrypt" (an improper terminology).
- Textbook RSA decryption becomes textbook RSA verification with message recovery less the necessary consistency check (or opening of the signature if we define textbook RSA signature as producing an appendix to the message). That's the case when selecting "RSA Key Type: Public Key", and "Encryption Algorithm: RSA" in the question's linked tool, then "Decrypt" (an improper terminology).
In best practices we don't perform 1 because it's pointless and can be insecure (as explained), and we don't perform 2 because
- An RSA public key and private key are NOT actually expressed as pairs of integers $(n,e)$ and $(n,d)$, thus are not interchangeable. That's because the conventional format for these keys are different (even the question's linked tool uses different format), and also the private key typically include $(n,e,d,d_p,d_q,q_\text{inv})$ for performance reasons, when only $(n,e)$ must be in the public key.
- Textbook RSA encryption is not used, because it is insecure and restricted to small message; and secure RSA encryption as practiced (e.g. RSA-KEM for key establishment, RSAES-OAEP for small messages including key transport, and hybrid RSA encryption) does not turn into working RSA signature.
- Textbook RSA signature is not used, because it is insecure and restricted to small messages; and secure RSA signature as practiced (e.g. RSASSA-PKCS1-v1_5 or RSASSA-PSS) does not turn into working RSA encryption.