# Authentication and encryption of messages

I have a hybrid scenario of public key cryptography in which messages between users are encrypted using AES with a symmetric key generated ad hoc.

A typical hybrid system scheme considers for every message to send the couple composed of:

• The message encrypted with the symmetric key
• The symmetric key (using AES GCM I suppose I have to include also the IV used and additional data? - And what additional data could I use?) encrypted with the recipient's public key

I want to sign the messages in order to authenticate the sender and - if possible - to authenticate also the data.

I want to ask you if in my scenario would it be better if:

1) I authenticate the message in plaintext enciphering it with the sender private key, then encipher the result (x) with the AES KEY obtaining y and lastly send the pair (AES key enciphered with recipient's public key; y)

2) I encipher the message with the AES key, then I encipher the AES key with the recipient's public key obtaining x and then I apply my private key to the pair (symmetric encyphered message, x). Finally I send the result.

I also checked the answers to this question in order to understand better this topic: Should we sign-then-encrypt, or encrypt-then-sign?

In order to allow data authentication should I add to one of the above schemes a MAC? How?

And, finally, does AES GCM provide for authentication in itself without other constructs to work with?

First and foremost, you don't mention why you feel the need to invent your own crypto protocol. "Don't roll your own crypto" is a common saying around here for a reason: there are so many small things to be aware of when implementing a crypto protocol, and if you do any one of them wrong, your entire protocol will be insecure. Designing crypto protocols really needs to be done by committees of experts - preferably in an RFC to give the public a chance to point out things that you missed.

For the use-cases you describe, why are you not using an off-the-shelf SSL/TLS or PGP library?

If this is just for your own learning, then cool, but if you plan to use this for real then please tell me the name of the project or product it will be in so I can, umm, take a closer look -_-

The following assumes that this is just a learning project.

You keep saying

encipher with the private key

What does that mean? That doesn't really make sense as a concept. I know that with RSA the public key and private key have the same format and can both be used for either encipherment or decryption, but in general with other algorithms (DSA, ECC, many of the post-quantum proposals) the two keys look very different and only the public key can be used for encipherment.

Are you trying to do a digital signature? If so, you need to calculate the signature for the massage, and attach the signature string to the packet in some way.

At it's core, your question seems to boil down to Should we sign-then-encrypt, or encrypt-then-sign? I'm not really clear what additional information you're asking for, but let's look at how some standard protocols solve this problem:

## TLS / SSL

TLS takes a different approach by using authenticaled key exchange prior to sending any messaged: during the initial handshake when the connection is opened, both parties use public-key crypto to authenticate themselves, and to establish a shared AES key that only those two people know. Once that has been done, you don't need to use public-key to authenticate each message because if the message decrypts properly with the shared AES key, then it must have been sent by the other person.

## PGP

This is a little closer to your use-case. You can read about all the packet types that PGP supports in section 5 of the PGP RFC-4880. My understanding is that the programmer or end-user has the choice of whether they do

sign-then-encrypt (like your option 1)

Symmetric-Key Encrypted Session Key Packet (Tag 3)
Signature Packet (Tag 2)
data


or encrypt-then-sign (like your option 2)

Signature Packet (Tag 2)
Symmetric-Key Encrypted Session Key Packet (Tag 3)
data


So this doesn't really help with your core question of sign-then-encrypt vs encrypt-then-sign.

## Bottom line:

1. You've asked a lot of questions here, and I have only answered some of them. I suggest you go back and study the fundamentals of cryptography. I think some of your questions would answer themselves.

2. For any practical use, please for the love of God, use a standard library of either TLS or PGP.

• First of all I thank you for your answer. Then, I know the common saying you quoted and - of course - my project is for a learning purpose only. Thinking about my question after some time I posted it I realized by myself that there were some errors, and others may surely be imputed to my lack of experience in the field (I know some of the fundamentals of the cryptogaphy but I'm aware there is still a lot to learn) and maybe to the haste I had in the translation in English because I wanted to have some useful asnwers. For all of this, excuse me. In essence, my project is very similar to PGP. – M-elman Dec 5 '16 at 18:26
• I decided to use the hybrid scheme I described in my question and I think it works for encryption. However, when I tried to figure out how to add the possibility of authentication, I got stuck because I couldn't understand how to give the user the chance to authenticate himself (and then I thought also to the possibility of data autenthication). Now, as I said in the previous comment, thinking again about my question, I realized that in the hybrid scheme, signing a message could be more complex than expected. – M-elman Dec 5 '16 at 18:36
• Sincerely, I knew that asymmetryc cryptography could be used for encryption or for authentication but I only knew the basic scheme for the latter, i.e. "To sign a message, m, the signer computes a signature, σ, such that σ ≡ m^d (mod N). To verify, the receiver checks that σ^e ≡ m (mod N)." As soon as possibile I will analyze better the digital signature in order to understand if I can apply it to my project in a easier way than that I was thinking about and that it was creating me these problems. Anyway, in the hybryd scheme I decribed, do you suggest to sign the message and then encrypt all? – M-elman Dec 5 '16 at 18:46
• @M-elman umm, sure? Let's take a giant step backwards: from that equation, I see that you are using RSA as your encryption / signing algorithm. Are you implementing RSA from scratch or using a crypto library / toolkit? I ask because all of this has been solved before, you are very much re-inventing the wheel. – Mike Ounsworth Dec 5 '16 at 19:29
• Your questions are very hard to answer because you are asking about many different things at the same time, and it doesn't seem like you have much understanding of the background material. I would suggest that you open several new questions on this site: rather than asking about your whole project, ask much smaller questions about individual pieces of theory like "How to compute an RSA signature?" or "How to use a MAC in authentication?". Many small questions are easier to answer than one big one. – Mike Ounsworth Dec 5 '16 at 19:34