In TLS, asymmetric encryption is used to agree on a symmetric key, which is used to encrypt/decrypt the data.

I want to ask if there is any scenario or system where the public / private key is directly used to encrypt and decrypt data.

  • $\begingroup$ If you use key for data, how would you recover your data? Do you have another method for decryption? $\endgroup$ – Cloud Cho Jul 31 '20 at 22:35

TL,DR: It's technically possible, but not in well-designed systems.

No, public-key encryption is never used to encrypt end-user data directly, either in TLS or in any other sensible protocol. Public-key encryption has many drawbacks compared to symmetric encryption:

  • It's a lot slower. A lot slower.
  • It can only encrypt very small payloads (a few tens of bytes). To encrypt more data would require a mode of operation, which carries its own share of difficulty (modes for symmetric encryption don't necessarily work automatically with public-key encryption).
  • If done wrong (e.g. PKCS#1 v1.5), it's vulnerable to oracle attacks.
  • It's more complex to implement, and at more risk of side channel leaks.
  • A successful quantum computer as currently envisioned would break all current asymmetric cryptography, but would only have a minor effect on symmetric cryptography (requiring twice the key size).

Sensible protocols are hybrid: they use asymmetric cryptography to establish a session key, and then use symmetric cryptography with the session key. Encrypting the session key is not the best way to do this. Rather than generate a session key and use public-key encryption to encrypt it, it's better to use asymmetric cryptography to generate a random session key. This is usually called a key establishment method (KEM) (although terminology can vary). The basic type of two-party KEM is key agreement, where each side uses its own private key and the other side's public key to generate a session key, in such a way that both sides obtain the same session key. Diffie-Hellman is the classic key agreement protocol. There are also one-sided KEM, for example RSA-KEM has one side apply the public-key operation to a random number in the correct range, and the other side decrypts it with the private key.

Good protocols use signature and key establishment in preference to public-key encryption. Signature and key establishment use the public key in combination with data received from an untrusted peer and use the private key in combination with data produced locally. In contrast, public-key encryption uses the private key in combination with data received from an untrusted peer, which gives an attacker more direct access to the most sensitive part of the system.

TLS may or may not use public-key encryption, depending on the cipher suite.

Most TLS cipher suites use a key agreement method (Diffie-Hellman, usually on an elliptic curve nowadays) to allow the two parties to exchange a symmetric key, and the client checks a signature made by the server to make sure that it's connecting to the expected server. With ephemeral Diffie-Hellman (each side generates a new DH key on each connection), this has the benefit of providing forward secrecy: an attacker who records a TLS connection and later manages to compromise one of the parties will not be able to decrypt the connection, because that requires the knowledge of the ephemeral key.

There is a set of TLS cipher suites that use public-key encryption and not signature or key agreement. You can spot them because they have “RSA” in their name, but not “(EC)DH(E)”. The client creates a symmetric key and encrypts it with the server's public key. In theory, this is secure: without the server's private key, an attacker cannot obtain the symmetric key. In practice, this method has historically been subject to many implementation flaws. And even with a perfect implementation, relying on public-key encryption lacks forward secrecy: if an attacker records traffic and later breaches a server, the attacker can use the server's private key to decrypt past connections. The most recent version of the TLS protocol (TLS 1.3) does not include any cipher suite based on public-key encryption.

RSA-KEM is not used in TLS.

Finally, to answer the literal question in the title, there are of course plenty of systems with badly designed cryptography out there. As we've seen, there are sensible (but not ideal) protocols that encrypt a session key. The asymmetric encryption mechanism doesn't care that what it's encrypting is a key, so it is possible to have a system that uses a public key to encrypt user data directly and that is functionally correct (i.e. what you decrypt with the private key is what was originally encrypted with the public key). If the size of the data to encrypt is small enough, the system even has a chance of being secure (i.e. the only way to decrypt data is to already have the private key). Due to the size limitation, this is effectively never done in sensible protocols.

  • $\begingroup$ Downvoted: while your points are reasonable, OP's question says nothing about "sensible". $\endgroup$ – Mark Morgan Lloyd Aug 1 '20 at 21:14
  • 2
    $\begingroup$ @MarkMorganLloyd ??? The question asks whether a public key is ever used to encrypt data directly, especially in the context of TLS. I answer that 1. sensible protocols don't do it and 2. TLS doesn't do it. Which part of my answer do you disagree with? $\endgroup$ – Gilles 'SO- stop being evil' Aug 1 '20 at 22:58
  • $\begingroup$ @Gilles 'SO- stop being evil' The question asks whether a public key is ever used to encrypt data directly, not whether sensible protocols only do it. I'm sure at least one proprietary crap product uses RSA as its only encryption somewhere, but I don't have a ready proof of that, because I tend to dismiss such products as insecure and unworthy of attention beyond advice not to use them. I upvoted: your answer is probably wrong, but it's more useful than a correct answer. $\endgroup$ – SAI Peregrinus Aug 5 '20 at 22:49
  • $\begingroup$ @SAIPeregrinus I still don't get what you think is wrong in my answer. Of course there's plenty of badly done crypto around there. That's why I don't say that it's never done, only that it's never done in any sensible protocol. Do you think that some of this “proprietary crap” counts as using a “sensible protocol”? $\endgroup$ – Gilles 'SO- stop being evil' Aug 6 '20 at 6:48
  • 1
    $\begingroup$ @user93353 Either this (e.g. with RSA-OAEP), or use the public key to generate a symmetric key (e.g. with ECIES or RSA-KEM) and encrypt the message with this symmetric key. It's called hybrid cryptography. $\endgroup$ – Gilles 'SO- stop being evil' Aug 9 '20 at 9:34

There's only a single case where I know this is done in practice, and it's for unattended sensing, but in a maritime context. You have remote sensors that have the ability to send data when a target is found, but due to the location of the sensor, you have a very low throughput transceiver . Typically, you get about 16-bytes per minute, which means that it does not matter about the speed decrease of asymmetric encryption. The packet is 8-bytes, and that all of the data that you send until you get your next target.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.