I build the computer appliance system that consist of software server and multiple controllers. The system I want to build must be pluggable in the sense that when I connect a controller C to the network, the server S will immediately finds it after initial message exchange(Step 1). After an initiation phase both C and S start TCP servers and able to send and receive messages to each other during multiple sessions (Step 2) without need for repeating Step 1 between sessions.

And now several challenges that I should to resolve

  1. Prevent spoofing of C and/or S on Step 1
  2. Prevent spoofing of C and/or S on Step 2
  3. Prevent replay attack on Step 2
  4. Ensure message confidentiality on Step 2

And here are some thougts about it:

  1. I should authenticate both sides on Step 1 to solve the first problem.
  2. I should perform the key exchange on Step 1
  3. I should encrypt all messages on Step 2 with symmetric key that have been received while Step 1 key exchange to solve the fourth problem.
  4. I should use nonces or timestamps on Step 2 to solve the second and third problems

Finally I want to ask some questions:

  • Are my thoughts about solving the problems right?
  • What is the best protocol for performing both authentication and key exchange at the same-time?

EDIT: The messages sended infrequently, so I need a lightweight protocol on Step 2. The protocol must be stateful of course to store symmetric key and possibly other auth info between sessions.

  • 1
    $\begingroup$ What about using a standardized protocol like TLS? $\endgroup$ Oct 31, 2012 at 19:06
  • 3
    $\begingroup$ What distinguishes a real C from a spoofed C ? $\:$ $\endgroup$
    – user991
    Oct 31, 2012 at 19:51
  • $\begingroup$ Sounds like a question that's better-suited for the IT Security site. $\endgroup$
    – D.W.
    Nov 1, 2012 at 0:44
  • $\begingroup$ @PaŭloEbermann, my messages sended one-by-one and infrequently so every time I need to send a message, C ans S must do the handshake, send the payload and break the connection, so if a payload is a few bytes, the TLS is too cost for me. $\endgroup$
    – tsionyx
    Nov 1, 2012 at 8:28