EpochLex entry

Tokenization

toh-kuh-nuh-ZAY-shun

The process of converting text or other input into tokens that a model can represent and process, often by splitting text into words, subwords, characters, bytes, or other learned units and mapping them to token IDs.