I am working on a system that includes our dear friends Alice, Bob, and a server that is only partially trusted. I have spent the majority of the day reading up on various cryptography methods, but I have concluded that I need some help to decide on the ones that best fit my use case.
For starters, I do not need authentication in the traditional sense. Alice needs to be able to communicate with the server securely, and for this Diffie-Hellman combined with public/private keys should suffice. In this case, the server is of course trusted.
Secondly, Alice needs to be able to communicate with Bob, via the server, and in this case, the server is not trusted. Therefore, another set of DH+keys is generated for that particular line of communications.
Now, the reason why I do not need authentication, is because this is all up to the server. The public key offered by A and B to the server will act as their temporary identity. This identity may or may not be reused later, so it is preferrable to use stored keys. If they are unable to read packets encrypted with this public key, then their ID is not deemed valid by the server, and a new one is istablished. In this case, the server is trusted.
I was considering using TLS, but I am unsure as to how I would programatically get access to the public key used in the OpenSSL session, and rolling my own TLS implementation seems like reinventing the wheel.
So what it then boils down to is this question: Which methods of cryptography should I go for that allows easy implementation of the communication scenarios above?