Tokenizer
Definition & Meaning
Part of speech noun
Quick meaning: Tokenizer is a noun meaning (computing) A system that parses an input stream into its component tokens.
Meaning
(computing) A system that parses an input stream into its component tokens.
Word Forms
- tokenizers
Related Words
Related dictionary entries include tokeniser, parser, terp, serializer, bracketer, detokenizer.
Related Dictionary Tools
Sources & Attribution
FreeDictionaryAPI.com / Wiktionary · Original source · CC BY-SA 4.0
Dictionary data is provided by FreeDictionaryAPI.com and sourced from Wiktionary under its stated license.