What is the best algorithm to encrypt files for storage on untrusted servers?

I need an encryption algorithm that works fast on large files and doesn't increase the file size. It should use a key to encrypt/decrypt data. The files will be sent using ReST and over HTTPS, but a good encryption/decryption method with a key is mandatory (to be used in a website).

The files will be saved on a 3rd party servers, and although the files are sent through https, they must be stored encrypted on the remote servers and decrypted when retrieved. The key will not be distributed. The key will be stored on my trusted server, as it is also the receiver! I want to encrypt the file, put it on the 3rd party server, then when I need it I can retrieve it and decrypt it.

-
You are missing some essential details. What type of file, how are keys exchanged, etc? The best advice I can give is to stick with a standard. In this case OpenSSL or SFTP would be good choices depending on the answers to the questions above. –  mikeazo Oct 1 '12 at 12:40
Unless you at least know the information necessary to make a decision like that, you should hire someone to help you make that decision. From the way this question is asked, we can't even tell the most basic things about the type of encryption you need. –  David Schwartz Oct 1 '12 at 13:26
the files will be sent using ReST and over HTTPS , but still a good encryption/decryption method with a key is mandatory (to be used in a website) –  Ronan Dejhero Oct 1 '12 at 13:33
@RonanDejhero, I'm thinking your question would be better off on our sister site IT Security. Would you like me to move it there? –  mikeazo Oct 1 '12 at 13:53
@RonanDejhero, HTTPS traffic is encrypted. Why do you need more than that? –  mikeazo Oct 1 '12 at 14:08

It seems your major requirement is that the file size not increase. This is possible, but at the sacrifice of some security (namely integrity and authentication), so it seems you will only be able to provide confidentiality. If the file size can increase slightly (say no more than 300 bits) you should encrypt with an authenticating mode such as GCM.

As Thomas points out, streaming modes will not alter the file size, so use one of those. There are some things you have to realize though if you go this route. Streaming modes are malleable. This means that an attacker (i.e., the 3rd party server storing the data) could make changes to the ciphertext which have a predictable change to the plaintext upon decryption.

In CFB, if the attacker flips one bit of the ciphertext, it will flip the corresponding bit of the plaintext, plus the next block will be completely scrambled. In OFB, if you flip one bit of the ciphertext, upon decryption, the corresponding plaintext bit will be flipped. Same with CTR, change one bit of ciphertext and you will flip the corresponding bit of plaintext.

This could have disasterous consequences. For example, if the attacker knows that the file is a text file of the form Transfer 500.00 dollars to mikeazo, the attacker (me in this case) could change that to say Transfer 50,000 dollars to mikeazo.

Another thing to keep in mind is the IV or Nonce required by streaming modes. These have their own requirements and will have to be stored or communicated some how. They can also be tampered with potentially, which results in problems (e.g., perhaps only one block of ciphertext will be unrecoverable, or maybe the entire thing will be unrecoverable).

A final aspect to consider is keying. How will the key be distributed?

My point is, yes, you can use a streaming mode with AES to encrypt a file and not increase the file size. But, there is a plethora of unintended consequences that you (or whoever is mandating this) is not considering. If this is to be used in a production system for a business, you are putting yourselves at great risk, and if that is the case, hire a professional. It will save you money in the long run. If that is not an option, use well established standards/software. I'd recommend something like GPG if the licensing works for the application.

-
The key will not be distributed . the key will be only with the sender server , as it is also the receiver ! i want to encrypt the file then put it on the 3rd party server , so when i need it i get it back and decrypt it . –  Ronan Dejhero Oct 4 '12 at 12:54
@RonanDejhero Perhaps you should add this relevant detail and everything else of importance in your original post. You can edit it. –  Thomas Oct 4 '12 at 12:58

The question still lacks detail, so this answer will be equally vague and hand-waving.

Any modern cipher exceeds consumer disk/network IO speeds by a significant margin, you can pair any modern block cipher (AES is probably the best choice here) with a streaming mode of operation such that the file size is left intact, such as CTR or CFB. However, you still need integrity on top, so you also want a fast and strong hash function to use as a HMAC (or use AES in an authentication-capable mode, but such modes are not very widespread yet). A good hash function would be SHA256 (reasonably fast).

There is no "best" algorithm, there are only those that meet your needs and those that don't. Your needs are security and speed - choose accordingly. I gave some fairly generic suggestions above.

Of course, the better choice would still be to use something that's already production-ready, such as OpenSSL. This way you don't need to implement it yourself and just have to learn the API and make a few function calls at either end of your network sockets, and you don't have to worry about security problems (or at least, not as much).

Again, you should not be implementing cryptography if you don't know what you are doing, so your comments such as "I wish I could know that by my self" are worrying to say the least...

-
+1. This problem is solved. Use SSL. –  Stephen Touset Oct 4 '12 at 18:17
@StephenTouset SSL requires trusting the server. While the question is pretty vague, I suspect this requires host proof security. –  CodesInChaos Oct 7 '12 at 13:41
Sorry, I had meant to say *Open*SSL. –  Stephen Touset Oct 7 '12 at 16:52