Consider the following two authentication schemes. In both cases the server knows a public key of the client (for a signature system in the first and an encryption system in the second case).

Scheme A

Server presents a randomly generated plaintext to the client, and asks the client to sign the message. The server can verify the signature using the public key.

Scheme B

Server randomly generates a plaintext, encrypts the message using the client's public key, and asks the client to decrypt and send back the plaintext. The server can verify that the plaintext matches the one it generated.

The second scenario feels less secure than the first scenario, but I cannot quite put my finger on why.

Is there any effective difference between the two authentication schemes?


2 Answers 2


As long as properly implemented using secure algorithms, there is no real security difference. In both cases the protocol is secure as long as the underlying signature or encryption algorithm is.

However, one difference is the random number used:

  • In scheme A the numbers must be unique. If the server ever reuses a number, then an adversary could replay a previous signature they have observed.

  • In scheme B the numbers have to be unpredictable, or an adversary can authenticate just by guessing correctly, but as long as the encryption algorithm is semantically secure, repeats are not a problem.

You need larger random numbers in the first scheme than the second, due to the uniqueness requirement.

As mentioned in the other answers, this is not secure against man in the middle attacks. It could be run in a channel that prevents those. For example, in a typical TLS situation the client would have authenticated the server already, man-in-the-middle attacks would be ruled out, and the server could use either of these to authenticate the client.

  • $\begingroup$ In scheme B the numbers have to be unique as well. Otherwise the attacker knows the plaintext of the encrypted message, as he has seen the encrypted message + answer before. $\endgroup$
    – Exac
    Oct 19, 2015 at 11:00
  • 1
    $\begingroup$ @Exac with a semantically secure encryption algorithm, seeing the encryption of one number should not help the adversary determine whether another ciphertext also encrypts the same message. $\endgroup$
    – otus
    Oct 19, 2015 at 11:03
  • $\begingroup$ You are absolutely right. Just mentioned it $\endgroup$
    – Exac
    Oct 19, 2015 at 11:05

Neither scheme, as described, is secure against man-in-the-middle attacks. In particular, an attacker who wants to impersonate the client can simply relay the server's challenge to the real client, pretending to be the server, and then relay the real client's reply to the server. After that, they're free to keep pretending to be the client.

However, scheme B can be rather trivially modified into a simple key-exchange protocol that does protect against such an attack. Specifically, instead of sending the random number back to the server in the plain, the client should use it (or something derived from it, using a KDF) as the key for a symmetric (authenticated) encryption scheme, and encrypt all subsequent communications with the server using that key. Thus, the server can be confident that any messages encrypted with this key really come from the same client that successfully decrypted the original encrypted key — effectively, the two parties have established a secure channel.

Of course, the scheme still only authenticates the client to the server, but not the server to the client.* A simple way to fix this would be to provide the server with a private signing key, whose corresponding public key is known to the client, and have the server sign the initial message after** encrypting it with the client's public key.

Alternatively, you could have both the client and the server generate a random secret number, encrypt them with each other's public key, exchange them, and then hash the two numbers together to derive a shared key for symmetric encryption. This takes an extra message from the client to the server, but avoids the need for signatures. And you can eliminate the extra message overhead by cleverly combining messages: as soon as one party has received the other's random number, they can combine it with their own random number to generate the symmetric key, and start using it to transmit actual data even in the same message that also carries their own random number.

(Yet another option, if you want to have a mutually authenticated secret channel using only signature keys, would be to use something like Diffie–Hellman to first establish an anonymous secure channel, and then have both parties sign (a hash of) the D-H shared secret and exchange the signatures, thereby confirming both their identity, and also the fact that they're really talking over the same encrypted channel, with no middle-man attacker in between.)

*) I'm following your usage of the names "client" and "server" here, even though they're kind of backwards from the usual case. Typically, it's the client that initiates the communication, and knows the server's public key, rather than vice versa.
**) The reason why you need to sign after encryption is to prevent another MitM attack, where the attacker impersonates the server by first connecting to the real server as a client, obtaining a signed key, decrypting it and re-encrypting it with the target client's public key, and passing it on to the target client.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.