Tokenization is the process of breaking a stream of text up into words, phrases, symbols, or other meaningful elements called tokens.

learn more… | top users | synonyms

4
votes
3answers
383 views

How feasible is word-level frequency analysis over English (or any language)?

Say I have some black box which, given any English word, deterministically outputs a token for that word. Assume our black box is implemented using strong cryptography, i.e. the hardness of reversing ...
0
votes
0answers
20 views

Secure algorithm for calculating token and comparison

I am trying to define algorithm that would create a irreversible token from IDs, which are 10 digit information, and then would be compared on two different end points. Between end points, only ...