Is there need for message authentication if a public key is transmitted and then used to encrypt a symmetric key?

Question

First I'd like to note that I'm really a beginner at cryptography, and while this interests me and while I have researched it in the past few months, there is still a lot of things that simply aren't clear to me, and I would like to have them clarified so I can move on.

Let's say (because this scenario is what I have pondered and have thought about it for a while) that a client has generated a temporary RSA key pair, knows a server's public RSA key beforehand and needs to transmit its own public key securely to the server, in order to agree on a mutual, secret symmetric key, resistant to both eavesdropping and MITM. (I'm not aware of another, better way of negotiating a key when a client has a server certificate beforehand, but that would have more to do with my crypto incompetence.)

So the client encodes its (n, e) in a series of bytes and encrypts the data using the server's public key taking care of proper padding (using, say, OAEP). The server receives this data, decrypts it, and then wants to send its part of the symmetric key to the client. So it chooses this random key, hashes it using a secure hash algorithm (say, SHA-2), encrypts it using the client's public RSA key and fires it off into the network along with the hash.

The client now receives a hash and RSA-encrypted data. It decrypts it using its private key, checking the padding, hashes it, and compares to see if the hashes match.

My question is, is there even need for the hash in the server response so the client knows it hasn't been MITM'd (because the server has proved it decrypted the client's public key)? Can an adversary which doesn't know the server's private key ever change the RSA ciphertext so it passes the OAEP check routines? Even if this happens in some way, the client will end up using a garbage symmetric key—is this dangerous i.e. could be exploited in some way? And if there is legitimate need for the client to know at this stage absolutely no one has tampered with the message, is my simple plaintext hash-based authentication secure? I feel it is pretty obvious but I haven't stumbled upon its use so far, so there may be a good reason. I've heard that MACs are a good, secure solution, but obviously there is no shared secret at this point in the negotiation so I don't see how they can be utilized.

Am I missing something painfully obvious?

Answer 1

Notice: This answer assumes the user doesn't encrypt the initial contact because I misread that part. Will update accordingly in due course.

---

Is there even need for the hash in the server response so the client knows it hasn't been MITM'd (because the server has proved it decrypted the client's public key)?

Sharing the hash of a key is a bit risky in general so I'd say shy away from doing that. In your case there's another reason why you shouldn't: You are correct about MITM, anyone can generate a key and send it back to the user.

The server must sign its response so the user is sure of its origin.

Can an adversary which doesn't know the server's private key ever change the RSA ciphertext so it passes the OAEP check routines?

The probability of that happening is negligible. Even if it does your security doesn't depend on OAEP's sanity checks, it depends on the strength of RSA. See next for details

[...] will end up using a garbage symmetric key—is this dangerous i.e. could be exploited in some way?

No. There are two ways to spot the problem: One is by signing the message - if there's a mismatch between the signature and the message, you know you can stop there.

If there's no signature, the plaintext should conform to some format (which could be a checksum encrypted along with the plaintext). If your application decrypts an altered message which does conform to the format, you're either very lucky or the app doesn't bother checking, which would make it absolutely worthless.

In your case, let's say the client gets a garbled symmetric key. The server (conforming to the protocol) would expect the client to say Hello! and instead would decrypt something along the lines of £Gh^T-31j3mqjgeY, which is a strong hint that something's amiss.

is my simple plaintext hash-based authentication secure?

No. See the answer to the first question. Anyone watching the user will see his public key coming along, generating a new keypair for every server won't help at all.