Take the 2-minute tour ×
Cryptography Stack Exchange is a question and answer site for software developers, mathematicians and others interested in cryptography. It's 100% free, no registration required.

In terms of the frequency of letters, how is it possible to have different frequent letters when the length of the text I'm analyzing is shorter?

At the moment, I'm comparing the frequencies of a long text and a subtext from that text. To my surprise, the most frequent letters changed. In the long one it was the letter e followed by the letter t, however in the small text it was t followed by e. Also, when I checked the frequency of different types of texts (e.g news articles), the frequency of letters also changed as well as the most frequent one.

The bottom line is, how can that be possible? It makes no sense to me.

share|improve this question
2  
There are two different effects: 1) For short texts the frequency won't be statistically significant unless you average over many independent samples 2) Different kinds of text use different words. For example the word frequencies you'd get from emails would be very different from those on wikipedia. It's very well possible that this shifts the frequency of letters somewhat. –  CodesInChaos Feb 28 at 16:43
1  
Consider how the opposite claim ("Letter frequency is always the same no matter the text length") would hold up if I gave you extracts of only 1 character... Letter frequency is heavily dependent on both source and length. –  figlesquidge Feb 28 at 17:10

3 Answers 3

up vote 5 down vote accepted

Letter frequency checks how often a letter appears in your text. Depending on your text, the frequency of letters will change. If your text is short and has the name "Alex" in it a lot, it will make the frequency of the letter X much higher than average. Generally when discussing the expected letter frequency, people refer to the average letter frequency that appears in the english language (http://en.wikipedia.org/wiki/Letter_frequency). Though for specific texts the fequency might not be close to the average, and with shorter texts the frequency might be dramatically different, overall the frequency of a long passage should be close to the average.

share|improve this answer
1  
You can generate your own frequency tables based on samples of related text. If you're decrypting data that came from Alex Corporation, build your sample from their public documents. If you're decoding a Shakespearean code, build your sample from the Bard's plays. –  John Deters Feb 28 at 18:09
    
Very true. Also a very good approach for more specific examples. –  Minkus CNB Feb 28 at 22:36

"In terms of the frequency of letters, how is it possible to have different frequent letters when the length of the text I'm analyzing is shorter?"

Well, if you analize the famous "The quick brown fox jumps over the lazy dog" you can't expect a frequency analysis to perfect match with big texts analysis.

It would be like making a Voting Intent Survey with only 5 people and expect to have the real statistics of your country. The sample must be large enough to apply assymptotic frequencies.

share|improve this answer
    
Right, thank you –  Sara Mar 2 at 4:24

Speaking in statistical terms, this is the difference between the law of large numbers and the "law of small numbers" (e.g. see Poisson distribution).

Short texts are not statistically significant, or more detailed: If you assume statistical independent letters (not true in general, but can be used as simplification), for short texts the variance will be much higher, so that you have to expect a larger gab between the expected value and your actual measurement.

If you want to know whether a sample coincides with a given frequency distribution, there are statistical hypothesis tests, e.g. the chi-squared test, where the result indicates how likely your sample matches the given distribution.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.