# Is symmetric key encrypted with server's public key secure

I'm trying to implement lightweight yet secure protocol for communicating two trusted parties with each other.

Consider following scheme:

Alice wants to establish secure two-sided channel with Bob.

Alice knows Bob's public key from reliable source, so she generates random symmetric key, encrypts it with Bob's public key and sends result to Bob through untrusted channel.

Bob receives message, decrypts it with his private key and uses symmetric key found inside to write back to Alice, so they both share the same symmetric key. Alice and Bob then use any authentication scheme to verify identities of each other.

In case Mallory intercepts Alice's message, she won't be able to obtain Alice's symmetric key, but may forge her own, as Bob's public key is known. However Mallory will be unable to authenticate her, so she'll fail.

Are there any security drawbacks in this approach? Initially I've thought about Alice generating pair of asymmetric keys and using symmetric session keys for each message (GPG-style), but former seems more resource efficient to me.

-
What identification scheme? Based on the supposed shared secret that they think they have? Then how does Bob distinguish Mallory from Alice, if the former does an active attack on both phases? –  Henno Brandsma Jun 8 '13 at 13:06
@HennoBrandsma, yes, in original draft authentication is based on shared secret, so if channel is secure, in most trivial case Alice could just send Bob plaintext password. Mallory does not know Alice's random symmetric session key, as it was encrypted for Bob only. If Mallory forges Alice request to hold her key instead of Alice's, Mallory won't be able to forward Bob's response back to Alice, as she does not know original key. Mallory is also unable to obtain shared secret told by Alice to Bob by listening, as it's encrypted by random symmetric key, which Mallory was unable to retrieve. –  modchan Jun 8 '13 at 14:30
If they already share a secret password, cannot they base their scheme on that? Like SRP? Why bother with the RSA? –  Henno Brandsma Jun 8 '13 at 14:34
When Alice first meets Bob, she has to tell him her random chosen persistent secret for Bob to recognize her later - like registration. Bob assumes nothing about Alice upon their first meet. Alice cannot use DH due to MITM threat, so she sends her secret encrypted by Bob's public key - and the only way Mallory could interfere with this is to "register" herself too, without getting access to Alice-Bob communication. They don't use the same session key derived from Alice's secret because I've thought it's safer to choose random key each time. –  modchan Jun 8 '13 at 14:56
First, the PKE may need to be IND-CCA1 or IND-CCA2, depending on whether its used by any other people and whether Bob will permanently abort on a single failed attempt. $\:$ Second, this is very much not forward-secure. $\;\;$ –  Ricky Demer Jun 8 '13 at 20:29

The first part of the protocol – e.g. alice choosing a random secret key and sending it public-key-encrypted to Bob – is in essence the same as the usual RSA key exchange used in SSL/TLS.

As you mentioned, Mallory can't read the key, but can start her own connection to Bob pretending to be Alice, so this part only provides authentication of Bob for Alice, not the other way around. (This is okay for the common use case of HTTPS, where we have one trusted server and lots of untrusted clients). But (assuming your symmetric encryption also includes some kind of MAC and you have some protocol measures against replay attacks, like message numbers) Bob knows that he speaks with the same entity during the whole conversation.

Mallory might be able to send messages in Alice's name (and receive the responses to these messages), but can't read messages sent by Alice or read the responses to these messages (if they arrive at Bob).

To add authentication for Alice, you add a second authentication step over the now encrypted channel. This can be whatever wanted, from a plain password to another public key signature from Alice. Both schemes are commonly used in SSH, by the way.

This looks secure, but there might be some details hidden in the details of your protocol.

As mentioned by Ricky Demer, when Bob's private key is revealed/broken for some reason, all the previous connections (which an Mallory might have recorded) can now be read by Mallory. (This is called the non-forward-secrecy.)

Using Diffie-Hellman (together with some kind of authentication, like signatures from both parties) or SRP solves this by making the session key unrecoverable from just the transcript and the long-term secrets. This is incorporated in both SSL and SSH protocols, too.

In general I would recommend to use an existing (and evaluated) protocol like SSL or SSH as the base of your communication. (You don't have to implement all options.)

-
Thank you for full and comprehensive answer! –  modchan Jun 10 '13 at 11:32