Tokenization is the process of breaking a stream of text up into words, phrases, symbols, or other meaningful elements called tokens.

learn more… | top users | synonyms

3
votes
3answers
203 views

How feasible is word-level frequency analysis over English (or any language)?

Say I have some black box which, given any English word, deterministically outputs a token for that word. Assume our black box is implemented using strong cryptography, i.e. the hardness of reversing ...