How many different ciphertexts in repeated substitution cipher?

Question

Consider an alphabet with 26 letters. Then a substitution cipher has $26!$ possible ciphertexts (with every letter substituted). What about repetive substitutions on a 5 letter word for example? How many different ciphertexts will result? Like $\sigma(a)=b, \sigma(b)=c, \sigma(c)=a$, then $abc \rightarrow bca \rightarrow cab \rightarrow abc$ so two ciphertexts for $|A|=3$.But what about generally?

Answer 1

What you are asking is how many times do you have to apply the substitution on a plaintext to arrive back at the plain text. One observation is that it will depend on the order of the substitution as an element of $S_{26}$, the permutation group on 26 elements. If your permutation has order $n$, than applying it $n$ times is the same as the identity permutation. For a given substitution, it is not hard to determine its order.

Your question is a little different. For a given plaintext, it could take fewer iterations to return back. You are interested in the permutation's action on a subset of elements and not the full set. It is going to have to do with the orbits of each plaintext letter under the permutation. So for a plaintext "abcde", how long does it take each of a, b, c, d, and e to cycle back? The answer is the LCM of those lengths. In particular, notice that the answer might be totally different for plaintext "fghij".

Answer 2

The maximum possible cycle length of a permutation of $n$ elements is given by Landau's function. For $n=26$ the answer is 1260