I am learning cryptography and here is what I understood so far:

Story. I decide to talk with my friend over unencrypted channel. We exchange public keys with each other so that we can encrypt text files (messages). Once other party encrypts the message with my public key (the public key I given to my friend) and sends that encrypted file to me, I can decrypt message with my private key.

Here is how I create my key pair.

openssl genpkey -out privkey.pem -algorithm rsa -pkeyopt rsa_keygen_bits:4096
openssl pkey -pubout -in privkey.pem -out pubkey.pub

Then I can give this pubkey.pub to my friend who can encrypt message for me and send ecrypted file to me. I try this myself:

openssl enc -e -aes-256-cbc -in text.txt -out message.xxx -pass file:./pubkey.pub

This worked OK, I created encrypred file. But I cannot decrypt it:

openssl enc -d -aes-256-cbc -in message.xxx -out decrypted.txt -pass file:./privkey.pem

bad decrypt
139927387231896:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:529:

I could decrypt with pub key but that is not what I was trying to achieve. What I would like to is implement simple practical example of Alice and Bob https://en.wikipedia.org/wiki/Public-key_cryptography It is all clear and easy in theory, but I could not find any easy and clear practical example like mine.

Added, list of files:

ls -alh
total 28K
drwxrwxr-x  2 user user 4.0K Sep 21 17:59 .
drwxrwxr-x 11 user user 4.0K Sep 21 16:08 ..
-rw-rw-r--  1 user user   64 Sep 21 17:59 decrypted.txt
-rw-rw-r--  1 user user   96 Sep 21 17:58 message.xxx
-rw-rw-r--  1 user user 3.2K Sep 21 17:08 privkey.pem
-rw-rw-r--  1 user user  800 Sep 21 17:08 pubkey.pub
-rw-rw-r--  1 user user   74 Sep 21 17:58 text.txt

You are trying to encrypt and decrypt with AES, which is a symmetric algorithm, using your RSA keys (which are meant to work with RSA, which is an asymmetric algorithm).

To use AES, both sides need to know the same secret key - there aren't separate public and private keys. The way you would usually do this is to choose a random secret key for AES (often called a data encryption key or DEK), encrypt the file using that key and the AES algorithm, then use RSA to encrypt that DEK using your friend's public key. You would give your friend both the encrypted document and the encrypted DEK. She would decrypt the DEK first using her RSA private key, then use the recovered DEK to decrypt the document. This technique is commonly called envelope encryption.

  • 1
    $\begingroup$ Agreeing with Bob, just adding a supplemental link that describes the so-called Digital Envelope hybrid cryptosystem with a diagram and text: garykessler.net/library/crypto.html#why3 $\endgroup$ – Dan Sep 21 '18 at 23:58

As Bob Wall explained, I was using module (enc) which is used for symmetric encryption. Here is solution:

openssl rsautl -encrypt -pubin -inkey pubkey.pub -in text.txt -out message.xxx
openssl rsautl -decrypt -inkey privkey.pem -in message.xxx -out decrypted.txt

While I was learning, this answer was not so obvious https://gist.github.com/dreikanter/c7e85598664901afae03fedff308736b


Not the answer you're looking for? Browse other questions tagged or ask your own question.