Is there a way to make RC4 (ARCFOUR) secure, or is it completely broken?

Question

I need a method to authenticate a process with another in order to establish interprocess communication between them, to prevent malicious processes from trying to hook onto the system. Currently I just send a password in plaintext between the two processes and the host process just performs a string comparison with its copy of the password, and will disconnect the connecting process if they don't match. It was fine when I was limited to connecting each process on localhost, but obviously this is not secure if someone just happens to have access to a network between each process and could sniff the packets. Do note that I only care about encryption for the password exchange and I don't really need it to secure the communication.

So basically I need a encryption algorithm that doesn't need to protect its contents well but does need to protect its key, and needs to be as fast as possible with minimal RAM usage and I/O overhead. Yes, data exchange is more CPU-bound than I/O bound since the two processes will be doing loads of calculations and they most likely will be connecting through localhost. Now, I came across RC4, which has a throughput of over twice that of AES-128, but it is evident that it has some weaknesses and I have a lot of questions about how to implement it in a secure way.

Well first question - it's in the title. Is RC4 completely broken and should be avoided? I hear a lot of "it is secure if properly used" and then a lot of "use of RC4 is discouraged."

I was planning to keep the plaintext password on both processes and have them SHA-224 hashed, and concatenate the hash with a 32-bit IV as a 256-bit key for RC4. Are there any security risks with this approach? Perhaps I could then hash the concatenated IV and hashed password with SHA-256? I'm well aware that WEP was cracked because it used a 24-bit IV, but a 32-bit IV has over four billion more unique values. Should I avoid incrementing the IV after each packet like WEP?

And when it comes to RC4-drop[n], what is a reasonable value for n? 256, 512, 768 bytes? How would you implement drop n on a library whose only public operation is encrypting a byte array - pass a byte array of length n after the engine is initialized with the key? Would that mean you would have to basically process n empty bytes after every time you change the IV (which happens after every packet)?

Sources: http://www.rsa.com/rsalabs/node.asp?id=2009

EDIT: after reading all these excellent responses, I realized how ignorant I sounded about the whole subject. I had no experience with cryptography before this day, and I only was able to read about RC4's risks from the limited Google results that I could barely follow. thank you to everyone who has been patient with me!

Correct me if I'm wrong, but I believe a better solution is to either use AES (which is not slower because re-keying RC4 after each packet is expensive) or an universal message authentication code. I can easily work with AES, but my understanding of MAC is a bit shaky. An IV is only secure if the update sequence is randomized and unpredictable, meaning I would have to send a 128 bit IV for every message that is sent, which can add a 400% overhead to the short, 4 byte packets that are commonly sent by my application. HMAC can add more than twice that amount if SHA-256 is used. I am going to have to investigate this matter further.

Answer 1

A few observations:

1. RC4 suffers from related key attacks. This means your idea of concatenating a 224 bit key and a 32 bit IV is not a good idea. You should rather use $\operatorname{SHA-256}(Key\mathbin\|IV)$

2. Remember that a (Key, IV) pair must not be reused, ever.
   A 32 bit IV can work if it's a counter, but IMO such a scheme is unnecessarily fragile. I'd rather generate a new, random 128 bit value for each encryption.

3. Even with dropping, RC4 is still a bit biased. This can allow password recovery if an attacker can observe a few billion communications.

4. You don't protect data integrity. For example an attacker might keep the encrypted password intact, but change the rest of the message. I don't understand your problem well enough to know if that's a problem.

5. There might be replay attacks, where an attacker repeats an earlier message fully or in part.

6. Your assumption that RC4 is faster than other ciphers isn't necessarily true. On many modern systems AES is really fast. There are also other modern stream ciphers, such as Salsa20 which are fast and much stronger than RC4.
   RC4 has costly initialization. So its only fast if you need to initialize it rarely. In particular if you reinitialize for each packet, its performance will certainly drop below AES.

7. Do you really need a cipher in the first place? You might be better off with a MAC. It seems like you're interested in authentication, not confidentiality. Universal MACs can be really fast.

8. User Passwords often have low entropy, so you need to strengthen them before using them as a key. Strengthening them is computationally expensive, and is done using KDFs, like scrypt or PBKDF2.
   Consider using a high entropy key instead of a password.

9. If you use a constant key hardcoded into an executable, those executables may never run on untrusted machines. Else an attacker could simply extract the key out of the executable.

In short:

1. The weaknesses in your protocol are likely much more severe than the weaknesses in RC4.
2. There is little reason to use RC4 in a new protocol. There are fast and strong alternatives.

Answer 2

No, RC4 is not completely broken. It is possible to use it properly. It's just not very likely that an average developer will do so.

RC4 is not a good choice for new systems. It is tricky to use properly. There are some serious pitfalls which, if you're not an expert cryptographer, can bite you in the butt. In fact, if you take a quick look in the mirror, I think you'll find a few bite marks in your own trousers. :-)

So, some advice, starting with the general and on to the specific:

Don't roll your own crypto stuff. Don't. Just don't. You're likely to get it wrong. Indeed, you did get it wrong. And while you can try to fix up the mistakes you made that you learn about here, the result will probably still be wrong in another way that you haven't thought of, because you aren't aware of the decades of lessons that the crypto community has learned about how to avoid security problems.

Instead of trying to design your own encryption program, you should use an existing well-vetted system. For example, I recommend GPG (or PGP) for file encryption, TLS (or SSL or SSH) for encrypting a connection, or Truecrypt (or Bitlocker or FileVault 2) for encrypting a hard disk, etc.

In other words, if you are asking "What crypto algorithm should I use in the program I'm writing to encrypt my data?", you are asking the wrong question, because in most cases, you shouldn't be writing a program to encrypt your data: you should be re-using existing well-vetted programs.

Don't use poor entropy keys. Your scheme derives the crypto key from a password. This has some severe security problems, and you should avoid doing so if at all possible. If you must do it, learn how to mitigate the risks.

If you used a well-vetted crypto program, odds are that you wouldn't need to worry about this: the designers would have taken care of this for you.

Don't forget to use message authentication. Using encryption without also using some form of message authentication -- as you are doing -- is almost always wrong. You need to use an authenticated encryption mode, or use a message authentication code (MAC) properly, to prevent tampering with the message.

If you used a well-vetted crypto program, odds are that you wouldn't need to worry about this: the designers would have taken care of this for you.

Don't use RC4 improperly. Your post is exactly why people say things like "Don't use RC4" and "Use of RC4 is discouraged". RC4 tempts people to use it in ways that are totally broken. You got tempted and made some of those standard mistakes. For instance, you planned to concatenate the key and the IV. Don't! That's massively insecure. That is exactly the mistake that WEP made, that makes WEP easy to crack. (You said "WEP was cracked because it used a 24-bit IV", but that's not accurate. Current WEP crackers don't rely upon the width of the IV at all.)

If you used a well-vetted crypto program, odds are that you wouldn't need to worry about this: the designers would have taken care of this for you.

Do I need to repeat it again? Use a well-vetted crypto program; don't try to design and write your own. Naah, I think you got the message by now. :-)

Answer 3

My answer provides some sort of fix to the original RC4 construction, and provides an answer to the OP better than just criticisms of RC4 without solutions. I call it RC4U.

THEORY

The original RC4 suffers two forms of weakness in it's pseudo random key stream. They are long term and short term biases.

The short term bias reflects the inadequacies of populating the state array from the key. RC4U eliminates the original KSA part totally. The state is generated directly from any key by a fully extended 8 bit Pearson hash producing a 256 byte derangement. This derangement becomes the state. My replacement should not be amenable to the development of Finney states.

Long term biases are extremely small. They are equivalent to biases seen in any decimal expansion of pi, and there is no practical attack using this vector. However, RC4U eliminates long term biases by appending a Von Neumann extractor directly after the PRGA component. The resultant output is therefore a bit stream, rather than a byte stream. Byte reassembly is required to use the output as a cipher.

PROS

• No short term bias

• No long term bias

• Any key of any length

• No padding required for key

• Likely that there are no Finney states

• Extremely simple to understand

• Extremely low resource requirements, suitable for Arduino /PIC

• Perhaps much faster than Arduino AES /ChaCha library

CONS

• Heath Robinson health working

• Requires a fixed 256 byte derangement array in the code

• Much slower than vanilla RC4

• Poor reputation

• Probably poor choice of name

This may be home brew crypto but we all know that high reputation members write it anyway. It is proven to work out to 40GB of output, and theory suggests it working further still.