# Frequency of letters change by the length of the texts?

In terms of the frequency of letters, how is it possible to have different frequent letters when the length of the text I'm analyzing is shorter?

At the moment, I'm comparing the frequencies of a long text and a subtext from that text. To my surprise, the most frequent letters changed. In the long one it was the letter e followed by the letter t, however in the small text it was t followed by e. Also, when I checked the frequency of different types of texts (e.g news articles), the frequency of letters also changed as well as the most frequent one.

The bottom line is, how can that be possible? It makes no sense to me.

-
There are two different effects: 1) For short texts the frequency won't be statistically significant unless you average over many independent samples 2) Different kinds of text use different words. For example the word frequencies you'd get from emails would be very different from those on wikipedia. It's very well possible that this shifts the frequency of letters somewhat. – CodesInChaos Feb 28 '14 at 16:43
Consider how the opposite claim ("Letter frequency is always the same no matter the text length") would hold up if I gave you extracts of only 1 character... Letter frequency is heavily dependent on both source and length. – figlesquidge Feb 28 '14 at 17:10

Letter frequency checks how often a letter appears in your text. Depending on your text, the frequency of letters will change. If your text is short and has the name "Alex" in it a lot, it will make the frequency of the letter X much higher than average. Generally when discussing the expected letter frequency, people refer to the average letter frequency that appears in the english language (http://en.wikipedia.org/wiki/Letter_frequency). Though for specific texts the fequency might not be close to the average, and with shorter texts the frequency might be dramatically different, overall the frequency of a long passage should be close to the average.

-
You can generate your own frequency tables based on samples of related text. If you're decrypting data that came from Alex Corporation, build your sample from their public documents. If you're decoding a Shakespearean code, build your sample from the Bard's plays. – John Deters Feb 28 '14 at 18:09
Very true. Also a very good approach for more specific examples. – Minkus CNB Feb 28 '14 at 22:36

"In terms of the frequency of letters, how is it possible to have different frequent letters when the length of the text I'm analyzing is shorter?"

Well, if you analize the famous "The quick brown fox jumps over the lazy dog" you can't expect a frequency analysis to perfect match with big texts analysis.

It would be like making a Voting Intent Survey with only 5 people and expect to have the real statistics of your country. The sample must be large enough to apply assymptotic frequencies.

-
Right, thank you – Scarl Mar 2 '14 at 4:24

Speaking in statistical terms, this is the difference between the law of large numbers and the "law of small numbers" (e.g. see Poisson distribution).

Short texts are not statistically significant, or more detailed: If you assume statistical independent letters (not true in general, but can be used as simplification), for short texts the variance will be much higher, so that you have to expect a larger gab between the expected value and your actual measurement.

If you want to know whether a sample coincides with a given frequency distribution, there are statistical hypothesis tests, e.g. the chi-squared test, where the result indicates how likely your sample matches the given distribution.

-