I'm currently building a website where the users would enter their credentials for another web service that I'm going to scrape to get their data.

I want to make sure that when I save their credentials in my database, I'm using the best encryption possible and the best architecture pattern to ensure the highest level of security.

The first idea that I had in mind was to encrypt the data using an RSA pub key (PBKDF2, PKCS1_OAEP, AES 256bit... ???) and then allowing my scrapping script to use the private key to decrypt the credentials and use them.

  • Could you recommend the right/best encryption to do this?

  • If my server is hacked, the hacker would have access to both the database and the private key, since it will be kept on my server that runs the scrapping script and hosts the DB. I know that it's a difficult equation to solve. But maybe there is an architecture pattern that helps with this.

I'm coding in python and I believe PyCrypto is the go-to library for encryption. (Sorry I have very little knowledge about cryptography so I might be confusing technologies)


I'm also looking to find out how to design my server(s) to secure all this. I know that if I use RSA, I need to put my private key somewhere where it won't be reachable from internet if my server is compromised, but still the private key needs to be reachable by my scrapping script that needs internet access.

So as you can see i'm a bit lost. I'm ready to use two independant servers but I don't have any idea about the best system design to use here.

Many thanks.

  • $\begingroup$ How much effort and / or money are you willing to invest into the security here? $\endgroup$
    – SEJPM
    Commented Feb 3, 2017 at 13:34
  • $\begingroup$ Hi @SEJPM, good question, it's a one man side project, i'm not building a critical piece of software for a bank. I just want that my users data get the best level of security that I can achieve as small time python dev. I just need a robust approach that I can implement in python. $\endgroup$
    – mandok
    Commented Feb 3, 2017 at 13:42
  • $\begingroup$ …then allowing my scrapping script to use the private key… – To clarify: you are planning to store that private key on the same (internet-connected) server you’re going to run the script on? If the answer is “yes”, how are you planning to handle potential server breaches which might give adversaries easy access to your private key (which – if you handle things the wrong way – could void most of your security efforts)? Therefore, it would be great if you could edit your question, describing that part of your plan too… thanks. $\endgroup$
    – e-sushi
    Commented Feb 3, 2017 at 14:11

1 Answer 1


The property you really want here is called Isolation.
You want all the three processes (Database, Scraper, Webserver) to run fully isolated from each other such that a compromise of the DB or the webserver doesn't end in a compromise of the passwords. So before we dive into how you can achieve this separation, let's just quickly assume we already established it and are bothered with the communication between the three modules.

Let's just quickly forget about the communication between the three modules except for when the passwords are involved (they're the most interesting part here). So you hand a public key to the webserver. Whenever a user enters his credentials for storage, you encrypt them using this public key and send the result to the database. Then your scraper will have the corresponding private key and fetch the encrypted credentials from the database, decrypt them and then fetch the data and store that in the database or send it to the webserver or whatever really.
As for the choice of primitives here: Use RSA-2048 with OAEP to encrypt a random AES key and IV and encrypt the credentials using this AES-GCM. Alternatively you may use RSA-OAEP directly on the data but this limits you to less than 200 bytes for the credentials. Alternatively you can consider using the OS's crypto services, a TPM, a secured SGX enclave or a proper HSM for private key storage.

But what does this fancy transport mechanism help you, when your webserver is compromised and an attacker can just read the private key out?
This is where isolation comes into play. Here are a few things you could do:

  • Clearly specify what queries the scraper and the webserver can make to the database and enforce the limitation from each individually to these queries using the database's credentials.
  • Consider logically or physically separating the services, ie by putting them all into different [chroot][8] jails, into separate VMs or running them on different machines.
  • Prefer separating the database server and the scraper script context-wise to ensure that a database server compromise doesn't lead to a credential compromise.
  • Depending on how often you restart the script, consider encrypting the private key of the scraper with a password, which you'd have to supply on every restart. To convert the password into a strong key, consider using Argon2d.
  • $\begingroup$ Many thanks @SEJPM, your answer is exactly what I was looking for. Using this blog post blog.brainattica.com/rsa-with-cryptography-python-library-2 approach will allow me to implement RSA-OAEP directly because i'm not constrained by the 200bytes limit. AES-GCM is available in the cryptography library cryptography.io/en/latest/hazmat/primitives/… The isolation approach is very nice. I will work on creating specific access for each component of my project using the DB's ACL. $\endgroup$
    – mandok
    Commented Feb 4, 2017 at 0:23
  • $\begingroup$ Hi @SEJPM, what do you mean by separating context-wise the database server and the scraper script ? Thank you. $\endgroup$
    – mandok
    Commented Feb 7, 2017 at 10:41
  • $\begingroup$ @mandok making them run in different security contexts (ie under different user accounts, in different VMs, ...) such that a compromise of let's say the DB doesn't also compromise the script (and thus the passwords). $\endgroup$
    – SEJPM
    Commented Feb 7, 2017 at 11:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.