Just in case this is your goal: if you want to examine the keystream of an arbitrary RC4 key, but all you have is a black-box library that takes a key and a message and encrypts (or decrypts) that message, you can get the keystream simply by passing a message composed entirely of zeros (null bytes). Any value XORed with 0 is just itself, and RC4 encryption (and decryption) are done by XORing the message with the keystream.
RC4 - like all stream ciphers - is essentially just a (theoretically cryprographically secure) pseudo-random number generator. The "seed" of the PRNG is just its key (in practice, usually a public portion to form an initialization vector / nonce that changes for each message, and a secret key that is pre-shared or exchanged via some other mechanism, concatenated together and possibly hashed). From such a "seed", the keystream is produced in a deterministic fashion, but the length of the key/seed is such that brute-forcing it is effectively impossible.
The actual algorithm by which the "seed" is processed and then used to generate bits (the keystream) is a sequence of basic mathematical operations, particularly addition, modulus, and bit swaps. The full algorithm is published on Wikipedia.
In theory, a secure PRNG / stream cipher has the following security features:
- Knowing only part of this seed (the IV, which is public, but not the secret key) does not allow recovery of the keystream with any likelihood greater than 1/(2^n), where n is the number of bits not known.
- Changing even a single bit of the "seed" - that is, even one bit of the key or IV - will produce a keystream that has no detectable relationship to any other (that is, it appears not only random by itself, but random compared to the output of any other seed).
- The keystream will not repeat (it contains no cycles, at least not within a length that could ever be computed or stored using current or forseeable-future hardware).
- Each bit of the keystream should appear totally independent of all other bits; you should be able to know every single proceeding bit of the keystream and still have no better than a 50/50 chance of guessing the next bit (or better than 1/256 chance of guessing the next byte [octet]). Same for knowing all the following bits, or indeed every single bit save one.
- Even knowing any practical number of bits of the keystream shouldn't allow anybody to predict the value of even a single bit of the "seed" with better than 50/50 chance.
Most non-cryptographically-secure PRNGs miss some or all of these properties (for example, they might loop after 2^32 bytes of output, or allow working forward or backward, deriving future or past outputs from some set). RC4, unfortunately, also fails to live up to these ideals; it was found to be vulnerable to multiple attacks that reveal at least partial message contents, or even the key, under common uses that occurred in the real world. For this reason, it is now a deprecated cipher / CSPRNG. If you want to examine a modern stream cipher / CSPRNG, consider the Salsa20/ChaCha family; as with RC4, the internal mathematical operations are fairly simple and described on the Wikipedia page (and, of course, in formal publications).