# Is this a sensible cryptographic protocol intending to reduce the impact of compromised security?

I'm in the process of designing a cryptographic protocol which will reduce the impact an attacker will have if they gained root access to a server storing data.

The basic crux of it is that keys which were used to encrypt data are not stored on the server and are only ever in memory for a short period of time.

The basic description is that:

• a Public/Private Key pair is generated on both client and server.
• The data is encrypted with both public keys by the client and transmitted to the server
• The server's private key is used to decrypt the data by one level (Still encrypted using client's pub) and stored on the server
• If the server needs to decrypt the data for ananlysis, this is initiated by the client who does this by sending their private key, encrypted by the server's public key (*This is the weakest point I think?)
• The client's private key is decrypted and use to decrypt the data (Then thrown away)
• Analysis results are then stored using client's public key (And optionally transmitted to the client)

A picture showing this can be seen :

Please be as brutal as you like - I'd like to try to get this right.

Also please note - I'm aware that if a baddie has root access to the box he'd be able to listen for the client's key transmission and decrypt that using the stored server private key and then compromise the data, but I'm trying to minimise the impact of server compromise, not remove it completely. I want to work with the assumption, that a hack would be detected before they had time to wait for a client to send their key.

EDIT: To Clarify - there's an assumption that the data can't be transmitted back tot he client for analysis - lets say because it's too large to make that efficient There's also an assumption that network traffic is monitored so an attacker would only ever have limited time on the server to attempt to steal the data.

-
Just a hint: Generally the basic idea is that you get cryptographic security by ensuring that the key never gets anywhere it might get compromised. Your protocol might give you security by obscurity at best, given your own assumptions about the threat scenario. –  Henrick Hellström May 16 '13 at 22:31
This sounds like a better fit for the IT Security site. It's not really about the cryptography per se, but about threat mitigation. If you click "flag" and ask the moderators to move it, they'll do so. –  D.W. May 16 '13 at 23:33
@D.W. I disagree - my question is about if there are any fundamental flaws with this cryptographic protocol as a principle. IT Security is much more about actual applications of protocols. –  Matt Fellows May 17 '13 at 7:08

I don't think the approach you sketched helps very much. If the server is compromised, the attacker can pretty easily modify the server-side software to log and record all the cryptographic keys, and then you haven't gained anything. Therefore, I don't think the approach you sketch is likely to be a great way to spend your limited software development budget.

If you want to mitigate the risks of server compromise, let me suggest two alternative approaches that I think might actually be worth your time:

• Privilege separation. Rather than have one server that does everything, split it up into two servers. One server does just the crypto decryption, and the other does all of the other application functionality. Install these on two separate servers, with separate passwords etc. so that if the latter is compromised, it won't compromise the former. The former server is super security-sensitive, but if you also keep its functionality very simple and if you administer it carefully, you may be able to reduce the likelihood of compromise of that server.

• Client-side cryptography. Don't ever allow the keys to be on the server. Do all of the encryption purely on the client side. The client generates the cryptographic keys, and never allows the secret key to leave the client (never shares it with the server); encryption or decryption with the private key is done entirely on the client side (not by the server). However, it takes additional work to make this work, and make it secure, and address the resulting key management and usability challenges, so this is not a no-brainer: it comes with some non-trivial disadvantages.

-
Thanks for the response: The assumption that the compromise wouldn't be permanent is made due to monitoring of all network traffic. Anything un-expected would be investigated, so logging and transmitting by a third party would never be kept up for very long. Privilege Separation is probably worth me looking into, but only using client side crypto is not, as we are talking terabyte datasets and huge numbers of distributed nodes doing the analysis. –  Matt Fellows May 17 '13 at 7:02
... so, your nodes are going to need to read more of that data than they $\hspace{1.96 in}$ have the ability to decrypt with AES? $\:$ –  Ricky Demer May 17 '13 at 7:35
@RickyDemer They read over a long time, and the data might need to be analyzed in a short time frame. –  Paŭlo Ebermann May 17 '13 at 18:48
"The assumption that the compromise wouldn't be permanent is made due to monitoring of all network traffic." - That makes no sense. It is not accurate or prudent to assume that monitoring of network traffic will detect all, or even most, of all compromises (or of instances of exfiltration of data by an attacker). I'm afraid your confidence in ability to detect compromises is sorely misplaced. Just look at some of the APT threats that have managed to penetrate systems and avoid detection for years. –  D.W. May 17 '13 at 20:50
@D.W. If the system is only ever intended to be accessed by limited IP addresses then any "foreign" IP addresses could be identified surely? Can you be more specific as to how this assumption is not valid? –  Matt Fellows May 20 '13 at 8:17
show 1 more comment