The normal method is to reverse-engineer whatever software produces or consumes the ciphertext. The underlying idea is that in a practical attack situation, the attacker has some known interest in breaking the system; there is context. The context usually indicates what kind of algorithm is in use.
In some cases where the code to reverse-engineer is not immediately apparent, other clues can be gathered. For instance, a smart card which can encrypt data at relatively high speed must necessarily use some dedicated hardware, so data sheets from the smart card vendor can point to what algorithms are possible.
Many communication protocols strive for algorithm agility and therefore include an explicit header about what algorithm is used. For instance, in SSL/TLS, the set of algorithms which will be used by client and server is negotiated in plain view, so what encryption algorithm is used is perfectly known to everybody, including any attacker.
Now, if you envision a formal, unrealistic setup where you are just given a black box which encrypts data with an algorithm and a key that you do not know, then guessing the algorithm details would amount as a break, in an academic point of view. Good encryption is supposed to be indistinguishable from randomness, which implies, in particular, that any secure encryption algorithm is supposed to be indistinguishable from every other secure algorithm.