This answer talks about how non-prime numbers would make the algorithm easier to break, not that the algorithm doesn't work full-stop: Why does RSA need p and q to be prime numbers?
Question 1: this implementation uses the number 57, which is not prime as it divides by 3.
- Choose p = 13, q = 57, e = 5 and message 'm' = 4
- Therefore n = pq = 741 and (p - 1)(q - 1) = 672
- encrypt message m as m^e mod n = 4^5 mod 741 = 283
- calculate decryption key d as e^-1 mod (p–1)(q–1) = 5^-1 mod 812 = 269
- decrypt '283' by 283^d mod n = 283^269 mod 741 = 199 ≠ 4
How does using a non-prime number cause the result to be wrong? I suspect the answer has something to do with the way that an inverse mod calculation looks for prime factors using the Euclidean algorithm, and if n is not the product of two prime numbers then it will give a different answer. But how is (p-1)(n-1) affected by p or q not being prime?
Question 2: p and q are the same prime number:
- Choose p = 11, q = 11, e = 3 and message 'm' = 2
- Therefore n = pq = 121 and (p - 1)(q - 1) = 100
- encrypt message m as m^e mod n = 2^2 mod 121 = 8
- calculate decryption key d as e^-1 mod (p–1)(q–1) = 5^-1 mod 100 = 67
- decrypt '8' by 8^d mod n = 8^67 mod 121 = 24 ≠ 4
How does using two (prime) identical numbers cause the result to be wrong? For this one I don't know where to start.