2
$\begingroup$

I'm trying to find faster ways to solve this: http://cryptogram.org/solve_cipher.html

Actually lists of common words are good, but they are often limited to 1000 words.

I found this, but I'm not sure which one to use, but the word list still seems too large.

Do you know lists of 10k, 20k, 30k words, or tools that sorts words by popularity?

Improve this question
$\endgroup$
2
  • 4
    $\begingroup$ List of words occuring in en.wikipedia ordered by popularity: the-interweb.com/serendipity/index.php?/archives/… $\endgroup$
    – CodesInChaos
    Apr 23, 2012 at 13:19
  • $\begingroup$ @CodeInChaos: Nice find. The Wikipedia list is also a good example of the fact that the source material does make a difference in word frequencies. I'm pretty sure that, for example, in almost any other corpus "median" would not be the 122-nd most common English word (and, in particular, it would not outrank "average", which is 142-nd in the Wikipedia list). $\endgroup$
    – Ilmari Karonen
    Apr 23, 2012 at 14:11

1 Answer 1

Reset to default
3
$\begingroup$

One nice way to generate a word list is distilling it from a large amount of texts.

There are a few interesting effects with this creation method:

  • The word frequency depends a lot on the chosen texts. For example when feeding it wikipedia the language is rather formal, and common every-day words are pretty rare. It might be interesting to compare Simple Wikipedia with the normal wikipedia
  • For some texts you also get common misspelling of words. That's nice for some uses, but bad if you want to use the list for spell checking. Should be fine in your case

I found a nice wordlist created from the english wikipedia which lists the words ordered by frequency. So you can truncate it if you want fewer words. Unfortunately it doesn't contain the frequency of each word, but since the source is available, adding this feature shouldn't be hard.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.