In RSA encryption as practiced (that is, to encipher a message which is a short symmetric key), the message size after padding is fixed and equal to the modulus size. Thus the size of the message has no impact on performance.
Calculating a modular inverse is performed only during key generation, that is seldom. Also, it has low cost compared to generating the primes and testing their primality.
The performance bottleneck (key generation aside) is typically modular exponentiation (or the few ones, when using the CRT for decryption). In typical implementations, exponentiation has cost $O(n^2)$ for encryption and $O(n^3)$ for decryption, where $n$ is the bit size of the public modulus. This can be lowered slightly using Karatsuba multiplication, or even more advanced techniques.
Update: there is simply no universally accepted way to encipher a message longer than the modulus size directly with RSA, thus asking "Theoretically, would the length of the message affect the performance?" is not well defined.
If we do a single RSA encryption, we have to make $n$ proportional to the message size $m$, thus cost increases as $O(m^2)$ for encryption and $O(m^3)$ for decryption, for typical implementations.
If we make $n$ constant and only use RSA, we must truncate the message into chunks separately RSA-enciphered, and for practical message size cost grows as $O(m)$ for both encryption an decryption.
Again the accepted/right practice is hybrid encryption, and there time spent doing RSA is independent of message size, with the overall cost $O(m)$ for symmetric encryption and decryption.