What tool is used to measure the security of rsa encryption? If no, is there any tool to measure rsa encryption in terms of security. Or any idea how to measure security of rsa encryption?
Strength of a security chain is that of its weakest link.
In RSA, one potentially weak link is factorization of the public modulus $N$ in the public key, an integer available to the attackers by hypothesis. The difficulty of factoring a properly chosen $N$ is extremely dependent on its bit size $n=\lfloor\log_2(N)\rfloor+1$, with the best known method GNFS. $n$ is often used as the primary metric of RSA security. $n=1024$ is considered a bare minimum nowadays (year 2018) even for low-stake applications (like DKIM, intended to prevent email spamming), with $n=2048$ in wide use. See this answer for an history of public academic factorization records. See the keylength website for recommendations on choosing $n$ for a new application.
Therefore, one primary tool to measure the security of RSA encryption is one that measures $n$, e.g. from a public key or cryptogram. Depending on context, that can go from "get info" on a file with the public key, an OpenSSL command, an hex editor...
A lot of other factors influence security of RSA encryption. For example, if the computer used is rigged with a spyware, poof can go any security, with $n$ becoming essentially immaterial. If the padding scheme is poor, some attacks can be possible. Tools to measure that? Uh..
The key size.
The longer the key size, the longer it takes an attacker to factorize the public key into the two private keys.
1024-bit keys are still in use in some applications but these keys should be replaced with a 2048-bit key (most applications already use 2048-bit keys). 2048-bit keys are estimated to be very secure until 2030. After that everyone should increase the key size again (i. e. 4096-bit keys).
Have a look at this link: http://www.jscape.com/blog/should-i-start-using-4096-bit-rsa-keys