Is symmetric key encrypted with server's public key secure

Question

I'm trying to implement lightweight yet secure protocol for communicating two trusted parties with each other.

Consider following scheme:

Alice wants to establish secure two-sided channel with Bob.

Alice knows Bob's public key from reliable source, so she generates random symmetric key, encrypts it with Bob's public key and sends result to Bob through untrusted channel.

Bob receives message, decrypts it with his private key and uses symmetric key found inside to write back to Alice, so they both share the same symmetric key. Alice and Bob then use any authentication scheme to verify identities of each other.

In case Mallory intercepts Alice's message, she won't be able to obtain Alice's symmetric key, but may forge her own, as Bob's public key is known. However Mallory will be unable to authenticate her, so she'll fail.

Are there any security drawbacks in this approach? Initially I've thought about Alice generating pair of asymmetric keys and using symmetric session keys for each message (GPG-style), but former seems more resource efficient to me.

Answer 1

The first part of the protocol – e.g. alice choosing a random secret key and sending it public-key-encrypted to Bob – is in essence the same as the usual RSA key exchange used in SSL/TLS.

As you mentioned, Mallory can't read the key, but can start her own connection to Bob pretending to be Alice, so this part only provides authentication of Bob for Alice, not the other way around. (This is okay for the common use case of HTTPS, where we have one trusted server and lots of untrusted clients). But (assuming your symmetric encryption also includes some kind of MAC and you have some protocol measures against replay attacks, like message numbers) Bob knows that he speaks with the same entity during the whole conversation.

Mallory might be able to send messages in Alice's name (and receive the responses to these messages), but can't read messages sent by Alice or read the responses to these messages (if they arrive at Bob).

To add authentication for Alice, you add a second authentication step over the now encrypted channel. This can be whatever wanted, from a plain password to another public key signature from Alice. Both schemes are commonly used in SSH, by the way.

This looks secure, but there might be some details hidden in the details of your protocol.

As mentioned by Ricky Demer, when Bob's private key is revealed/broken for some reason, all the previous connections (which an Mallory might have recorded) can now be read by Mallory. (This is called the non-forward-secrecy.)

Using Diffie-Hellman (together with some kind of authentication, like signatures from both parties) or SRP solves this by making the session key unrecoverable from just the transcript and the long-term secrets. This is incorporated in both SSL and SSH protocols, too.

In general I would recommend to use an existing (and evaluated) protocol like SSL or SSH as the base of your communication. (You don't have to implement all options.)