Say I have some black box which, given any English word, deterministically outputs a token for that word. Assume our black box is implemented using strong cryptography, i.e. the hardness of reversing a token to its word is reducible to some known assumption.
Now, assume I have a document corpus where a document is some list of English words. I run every word in every document through my black box and create a new document set of tokenized documents. I then give the documents to some attacker who then carries out a ciphertext-only attack to try to guess what the documents say.
I'm curious as to how successful this attacker will be in recovering partial information about the documents. He'll try to use statistical attacks to fit the frequency curve of the tokens to the frequency curve of English words. This will allow him to guess the preimages of more frequent words with high confidence, but will he be able to guess less frequently-used words? Are there more advanced attacks he could use?