EpochLex entry

Batch Inference

batch IN-fer-ens

A mode of model inference that processes many inputs together, typically asynchronously and without a requirement for immediate per-request responses, so predictions can be generated efficiently at high throughput.