I think I understand what you're asking for. You're trying to learn how we know which algorithm was used, so we know how to attack it. That's a part of what is known as cryptanalysis, the task of breaking ciphers.
If you are using a standard computer protocol, the encryption algorithm is defined as a part of the protocol. The computers can't talk unless both know what algorithm is in use. No attempt is made to keep it secret.
Generally there are plenty of clues will tell you what's going on. If things are flowing through port 443, the attacker is going to assume it's SSL protocol, look for the header which indicates what algorithms are in use, and begin there.
(If you're reading a joke or a movie plot on-line, ROT-13 has traditionally been used to keep you from accidentally seeing the punchline or spoiler until you're ready. Thus, ROT-13 is actually a de facto standard.)
Even without knowledge of the protocol, though, there are usually other ways to figure out which algorithm was used. The old school tool is frequency analysis. The cryptanalyst (the guy whose job is to break the cryptography) might plot out the frequency of characters. Normal English text follows a distribution such that the letter E appears 12% of the time, T appears 9%, A, O, I, and N appear about 8% each, and so on. (The nonsense words ETAOIN SHRDLU represent the 12 most common letters in English in descending order.) If the distribution of letters in the ciphertext approximately follows this same distribution, the cryptanalyst can assume a transposition cipher was used to shuffle the plaintext letters. If a different set of letters follows the same distribution, it's likely that a simple substitution cipher was used. (ROT-13 follows this pattern, by the way, and can be spotted because the distribution will be RGNBVA FUEQYH.)
If the distribution is flatter, the cryptanalyst might try measuring the distribution for every other letter, or every third letter, or every fourth letter, etc. If the distribution suddenly follows that same 12,9,8,8,8,8 curve, then the cryptanalyst has discovered that a Vigenere cipher or polyalphabetic cipher might have been used.
Once the algorithm is known, the next task is finding the key. Frequency analysis can help with the simpler codes, but with the advent of computers a different attack becomes possible. A computer can quickly try every possible key, which is called a brute-force attack. The number of keys you would have to try is called the keyspace. The classical Caesar cipher is a rotation cipher with a key of 3 (shift every letter to the left by 3 places.) The keyspace of a Caesar cipher is 26^1 - 1 (there are 26 possible shifts, but cleartext is not considered encrypted), or just 25. Because these increase exponentially with the number of possible keys, and because brute-forcing is generally done with computers, it's common to refer to these with power of 2 exponent notation. The keysize is generally referred to by the number of bits. 2^5 = 32, which is close enough to 25, so we can say the keysize of a Caesar cipher is 5.
Because the algorithm and key of ROT-13 is known (the algorithm is ROTate, and the key is 13) the keyspace is 1, and the only possible result is to decrypt. Therefore the keysize of ROT13 is 2^(1-1) == 2^0 == 1.
Keysize is an upper boundary of cryptographic security for an algorithm, as it represents the number of tries using brute force. A 40 bit keysize (2^40) is easily brute forced on a home PC. DES, with a 56 bit key length, was brute forced in the 1990s on a special purpose-built computer (Deep Crack.) Keysize is an upper boundary only, because if an attacker can figure out a cryptographic weakness in the algorithm, they can use it to reduce the number of brute force attempts. 80 bits is right at the margin of what is considered safe - advances in computer power make brute forcing larger keys more and more possible.
(Please note that this explanation doesn't work the same for RSA keys, where factoring a public key is the method of brute force. Today, a 1024 bit key is right on the margin of safety, and is not expected to remain safe much longer.)
Cryptanalysis is a fascinating study, and is highly recommended for anyone interested in cryptography. Being able to break protocols or ciphers is the first step in understanding what is needed to make protocols and ciphers more secure.