I'm new to cryptography. I have a binary image with 64x64 pixel in size. I perform an encryption to the image using RSA Algorithm. I wanna ask something bothering me, why can I not use big prime numbers as a pair of key for the RSA algorithm? Thank you in advance.
Although it is theoretical possible to create two large primes and use that to establish the private and public key, there are real issues doing so. First of all, let's check how big a modulus you need, as that defines the size of the two primes.
Let's assume color, and use the RGB color space without gamma, each color having 8 bits of color depth. That means each pixel uses 3 bytes. So your bitmap takes $64 \cdot 64 \cdot 3 \cdot 8 = 98304$ bits. That's excluding overhead of both the image as well as the overhead required for the padding required for secure RSA encryption (both of which may be insignificant with these kind of values).
This requires you to find two primes of half the number of bits to create secure RSA, assuming that you use the regular RSA and not multi-prime RSA. Unfortunately, even a modern CPU can take time to calculate an 8K-bit modulus. The problem is that each prime number is now close to 49152 bits. The density of primes is much smaller for such large spaces. Furthermore, doing primality testing on that kind of values will take a lot of time. So the time to find the primes $p$ and $q$ grows very rapidly (a factor of 12 to 16 for each additional bit). So the time to generate a key pair that size is prohibitively large (and depends on chance as well).
Just type in
openssl genrsa 98304 on your computer to use it as a heater.
So key pair generation is a huge issue at these key sizes. Performing any modular exponentiation with these kind of exponents and modulus sizes will be a lesser-but-still-significant issue as well. Many cryptographic libraries will have maximum key sizes for these reasons (and others, such as limiting testing, memory usage etc.).
So what can you do? Well: you can simply split up your plaintext into blocks and encrypt each with RSA. This will still be slow and will expand your ciphertext significantly compared to the plaintext image size, as padding the blocks is required.
What most cryptographers do is to pair the RSA algorithm with a symmetric cipher such as AES. In that case you can generate a random AES key and use that to encrypt the image (using a secure mode of operation). Then the AES key itself can be encrypted using the RSA public key. This kind of scheme is called a hybrid cryptosystem.