I'm working on a communication protocol where a client sends encrypted data to a server. What happens is that in some point, the client creates a symmetric key used to create data digests through the HMACSHA256 algorithm. Then, the idea is that data is sent to the server along with its digest, so the server can calculate the digest on its own and compare it to the digest sent to check data integrity.

For this, the symmetric key created by the client has to be sent to the server in a secure way, for which the key is RSA encrypted with the public key of the server but additionally, this encrypted message has to be encrypted with the private key of the client. I know that this second encryption has no sense since encrypting data with the private key doesn't assure data confidentiality, but the protocol states that I perform this action just for academic purposes.

The RSA keys are 1024 bits long so I can't encrypt data larger than 117 bytes and the RSA encryption produces 128 bytes long messages, so the question is, if after the first encryption (encrypting the symmetric key with the server's public key with RSA) I get a 128 bytes long message, how can I use RSA to encrypt this output with the client's private key?

  • $\begingroup$ "encrypting" with an RSA private key is typically associated with signing. Maybe that's what is intended? $\endgroup$
    – Tim McLean
    Commented Oct 18, 2015 at 21:21
  • $\begingroup$ Related / half cross post: stackoverflow.com/a/33197435/589259 $\endgroup$
    – Maarten Bodewes
    Commented Oct 19, 2015 at 10:58
  • $\begingroup$ You apparently want to produce a cryptogram for the encrypted session key, using the server's public key; then sign that cryptogram using the client's private key. The Java API allows you to do that without issue; your problem is that you are trying to use the encryption API for the second part, when you want to use the signing API $\endgroup$
    – fgrieu
    Commented Oct 20, 2015 at 7:33

1 Answer 1


The system is probably meant to effectively use an "RSA signature" on top of RSA encryption. If you used raw RSA it could "work", but only if the client's modulus is larger than the server's – or you get lucky and the encrypted key is small enough. Since you are using a library that handles padding for you, it expects a small enough input that it can pad it to size for encryption.

If you can, you could instead use an actual RSA signature algorithm in the second step. This would let the server to verify who the encrypted key is from.

However, if the protocol is something you must use, you should double check that you have understood it correctly. As you have stated it, it seems broken.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.