Tokenization is the process of breaking a stream of text up into words, phrases, symbols, or other meaningful elements called tokens.

learn more… | top users | synonyms

4
votes
3answers
383 views

How feasible is word-level frequency analysis over English (or any language)?

Say I have some black box which, given any English word, deterministically outputs a token for that word. Assume our black box is implemented using strong cryptography, i.e. the hardness of reversing ...
1
vote
0answers
92 views

How do I verify a reset token if the token is hashed in DB?

I've been searching around the web for best practice how-to on secure password reset system (email with reset url), and the idea to hash the token in the database, which I did not implement at first, ...
0
votes
0answers
21 views

Secure algorithm for calculating token and comparison

I am trying to define algorithm that would create a irreversible token from IDs, which are 10 digit information, and then would be compared on two different end points. Between end points, only ...
0
votes
0answers
15 views

PAN authenticatable irreversible tokenization algorithm

I need to implement algorithm to create authenticatable irreversible tokens from PAN, without using any secrets. Is it possible? Is there any standards for that from associations? I was thinking ...