Encrypt large data with RSA through blocks in java

Question

I'm working on a communication protocol where a client sends encrypted data to a server. What happens is that in some point, the client creates a symmetric key used to create data digests through the HMACSHA256 algorithm. Then, the idea is that data is sent to the server along with its digest, so the server can calculate the digest on its own and compare it to the digest sent to check data integrity.

For this, the symmetric key created by the client has to be sent to the server in a secure way, for which the key is RSA encrypted with the public key of the server but additionally, this encrypted message has to be encrypted with the private key of the client. I know that this second encryption has no sense since encrypting data with the private key doesn't assure data confidentiality, but the protocol states that I perform this action just for academic purposes.

The RSA keys are 1024 bits long so I can't encrypt data larger than 117 bytes and the RSA encryption produces 128 bytes long messages, so the question is, if after the first encryption (encrypting the symmetric key with the server's public key with RSA) I get a 128 bytes long message, how can I use RSA to encrypt this output with the client's private key?

Answer 1

The system is probably meant to effectively use an "RSA signature" on top of RSA encryption. If you used raw RSA it could "work", but only if the client's modulus is larger than the server's – or you get lucky and the encrypted key is small enough. Since you are using a library that handles padding for you, it expects a small enough input that it can pad it to size for encryption.

If you can, you could instead use an actual RSA signature algorithm in the second step. This would let the server to verify who the encrypted key is from.

However, if the protocol is something you must use, you should double check that you have understood it correctly. As you have stated it, it seems broken.