EpochLex entry

Byte-Pair Encoding (BPE)

BYTE pair en-KOH-ding

A subword tokenization algorithm that builds a vocabulary by repeatedly merging frequently occurring adjacent symbols or token pairs, allowing common sequences to become single tokens while less common text remains split into smaller units.