I'm working on a 16-bit processor msp430 and I need to know what I need to study when it comes to key size efficiency of a public key cryptosystem.
Every public-key system that I have seen in the constrained space was based on elliptical curves instead of the classic RSA scheme. This is primarily because cycle time is not as important as memory footprint and the 160-bits in ECC is believe to be roughly the same strength of 2048-bit RSA.
There seems to be some specific papers relevant to the MSP430 already. If you wanted to explore the space, you should start by looking at the cost in memory of storing the long binary words and the cost of each operation in memory. This will give you a good feel for the total memory, but not the calculation time. You also will find that things will be much smaller if you do it in assembly.