Mistokenize
Definition, Meaning & Examples
Part of speech verb
Quick meaning: Mistokenize is a verb meaning To tokenize incorrectly.
Meaning
To tokenize incorrectly.
Example Sentences
Similarly, all other two-character atomic representations in SMILES are being mistokenized.
The sentences were tokenized into words using the regex tokenizer which avoided the problems of mistokenizing while using the default NLTK tokenizer.
C++ considers ::*, .* and ->* each to be a single token and a single operator. Some pre-Release 2.0 implementations mistokenize expressions involving pointer-to-pointer-to-member.
These were mostly proper names, such as Ronny Johnsen, or foreign language items such as ambre solaire (French) and fairie queene (Middle English), as well as a few misspelt or mistokenized items.
Word Forms
- mistokenizes
- mistokenizing
- mistokenized
Related Dictionary Tools
Sources & Attribution
FreeDictionaryAPI.com / Wiktionary · Original source · CC BY-SA 4.0
Dictionary data is provided by FreeDictionaryAPI.com and sourced from Wiktionary under its stated license.