EpochLex entry

Token

TOH-kuhn

A small unit of text processed by a language model. A token may represent an entire word, part of a word, punctuation, or another portion of text.