EpochLex entry
Time to First Token
time to first TOH-kun
An inference-latency metric measuring the elapsed time from submitting a generation request until the first output token is received.
EpochLex entry
time to first TOH-kun
An inference-latency metric measuring the elapsed time from submitting a generation request until the first output token is received.