Tokenize
Definition & Meaning
Pronunciation /ˈtoʊ.kən.aɪz/ · Part of speech verb
Quick meaning: Tokenize is a verb meaning (transitive, computing) To reduce to a token or set of tokens by lexical analysis.
Meaning
(transitive, computing) To reduce to a token or set of tokens by lexical analysis.
More Meanings
- (transitive, computing) To substitute sensitive data with meaningless placeholders.
- (transitive) To treat as a token minority.
Word Forms
- tokenizes
- tokenizing
- tokenized
- tokenise
Related Words
Related dictionary entries include tokenise, detokenize, disassociate, analyze, unpack, delexicalize.
Related Dictionary Tools
Sources & Attribution
FreeDictionaryAPI.com / Wiktionary · Original source · CC BY-SA 4.0
Dictionary data is provided by FreeDictionaryAPI.com and sourced from Wiktionary under its stated license.