I'm creating a prototype for a messaging queue with some custom logic (can't re-use any existing solution).

One requirement is to have all messages encrypted. There are three classes of nodes subscribing to the queue with the following permissions:

The first and obvious thought was to use an RSA private key for group 1. This would allow to encrypt with the private key and decrypt with the public key.

Group 2 would receive the corresponding public key and would be able to only decrypt the content.

Group 3 would have no key and just handles the messages without access to the content.

After doing some further reading, there seem to be a number of problems with this approach (apart from the simple fact that the keys are used in "reverse" in relation to the intended mechanism).

Now the question: Is there some common approach to get this pattern to work? I have searched, but so far have not found anything.

I will have access to someone with more experience in this matter at a later point, but for purposes of the prototype, I just want to get something done that won't make me look like an idiot later. ;-)

-
You should not be encrypting "with the private key". $\:$ See this question and this answer. $\;\;\;\;$ –  Ricky Demer Oct 15 '14 at 9:44
Thanks, that did look like the obvious choice, but a little research showed that it's not the way to go. –  DeathByPiano Oct 15 '14 at 23:43

Yes. The key for encrypting and decrypting the messages can (and should) be that of a symmetric algorithm, e.g. AES-CTR with suitably random or sequential IV, and known to groups 1 and 2. $\;$ Signing should be on the enciphered data, using an asymmetric signature algorithm (e.g. RSA with PKCS#1v2 signature padding), with the private key known to members of group 1, and the public key known to all (as implied by its name). $\;$ Everyone (except possibly from group 3) should verify signature of enciphered data manipulated. –  fgrieu Oct 15 '14 at 9:56
Also: because RSA allows fast signature verification, it is a good candidate for the signature from an efficiency standpoint if reading and forwarding with integrity verification largely predominates writing/signing. If signature size overhead is an issue, RSA signature with message recovery can help reduce the signature overhead (e.g. to 34 bytes with ISO/IEC 9796-2 scheme 3 using SHA-256, with encrypted message of at least 222 bytes and 2048-bit RSA). $\;$ AES-CTR encryption/decryption is fast, and its size overhead is limited to the IV (e.g. 10 bytes for a counter to $2^{80}$). –  fgrieu Oct 15 '14 at 10:10