EpochLex entry
Tokens per Second
TOH-kunz per SEK-und
An inference-throughput metric that measures how many output tokens are generated per second, either for an individual request or across a serving system depending on the measurement definition.
EpochLex entry
TOH-kunz per SEK-und
An inference-throughput metric that measures how many output tokens are generated per second, either for an individual request or across a serving system depending on the measurement definition.