While solving a CTF challenge online, I came across a situation where I needed to decrypt a '.enc' file that has been encrypted using someone's public key. The corresponding public key '.pem' file was provided to me. I solved almost all other CTF challenges except this one and it has been bugging me since. I solved the first part: I have been able to decode a hint that says: "The file file.enc could be decrypted with a private key. Using raw algorithm, every 64 bits chunk of data was encrypted at a time. You can do the same thing to decrypt using RSA."

Here's what I have tried:

Read many posts like this one online and tried to decrypt the '.enc' file: http://www.czeskis.com/random/openssl-encrypt-file.html

I tried the following command for decryption: openssl rsautl -decrypt -in public_key.pem -out key.bin. I got the this error:

 no keyfile specified
 unable to load Private Key

Yes, but that's because private key is not provided to me. If I try to use the public_key.pem as infile, I get this error:

unable to load Private Key
139914348455592:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: ANY PRIVATE KEY

Tried the tool 'openssl' in Linux in several ways to try to decrypt the file.

Here is the 3 files zipped in tarball, if anyone wants to analyse them: http://www.filedropper.com/chaltar

Could someone please advise me on how to decrypt the '.enc' file before I give up on this challenge? I have worked long and broken the first part of the challenge, now I feel I'm so close.


  • I know the tool 'openssl' will be used to solve this challenge.
  • Encryption: RSA
  • Files: Only three files, public_key.pem, file.enc, and a random file that I decrypted that provided the hint I wrote above.
  • $\begingroup$ You cannot decrypt the file without the private key. Therefore, you need to obtain the private key somehow. Yet you are only given the public key. Hint: How can you find the private key given the public key? $\endgroup$
    – Reid
    Commented Apr 5, 2015 at 18:30
  • 1
    $\begingroup$ You shouldn't destructively edit your posts like that. These questions are meant to serve as potentially useful references for other users. If you remove all details from your question, you destroy any existing content these future users might need. $\endgroup$
    – Reid
    Commented Apr 7, 2015 at 16:29
  • $\begingroup$ This looks a lot like this question: crypto.stackexchange.com/questions/24804/… $\endgroup$
    – Andrey
    Commented Apr 7, 2015 at 18:22
  • $\begingroup$ This question or should I say challenge is from my challenge site. One of the rules is that you shouldn't cheat and literally posting our files on filedropper for analysis is not cool. I asked him to remove the question. @andrey, yes, that question you mention is the same. $\endgroup$
    – user23758
    Commented Apr 7, 2015 at 19:06
  • $\begingroup$ @Net-Force I suggest you contact stackexchange to have the material removed. Otherwise, it is up to the community. $\endgroup$
    – mikeazo
    Commented Apr 7, 2015 at 19:29

1 Answer 1

openssl rsa -pubin -inform PEM -text -noout < public_key.pem
Public-Key: (64 bit)
Modulus: 16513720463601767803 (0xe52c8544a915157b)
Exponent: 65537 (0x10001)

The modulus is small enough that you can easily factor it

After finding the prime factors, you can calculate the private exponent

After you have the private exponent, you raise each 64-bit block of ciphertext to the private exponent mod the modulus.

  • $\begingroup$ No, you have to write a program to do so. It's a pretty basic program though. $\endgroup$ Commented Apr 6, 2015 at 5:14

Not the answer you're looking for? Browse other questions tagged or ask your own question.