Batching

Three knobs multiply.

Knob	What it scales
Max replicas	How many machines
concurrent_workers	How many loops share one process
batch_size	How many requests one loop runs together

Ten replicas, four workers, and a batch of eight is 320 requests in flight.

import surfra

@surfra.handler(
    batch_size=surfra.batch_size(1, 8),
    batch_wait=300,
    concurrent_workers=4,
)
def predict(requests: list[surfra.Request]):
    return [{"text": item.input["prompt"]} for item in requests]

batch_size(1, 8) is the inclusive range. The handler is called with a list of that length. Return a list of the same length. Return None to fail every request in the batch.

batch_wait is milliseconds. A full batch runs immediately. A shorter batch waits until the oldest request has been queued for that long. wait is the same argument.

Caution

When batch_size max is 1, the handler still receives one Request, not a list. Turn the max above 1 before switching the function to a list.

concurrent_workers starts that many threads in the process after warmup. Each thread registers its own worker id. On the first registration, Surfra raises the replica’s worker cap to this number, up to 64. CUDA releases the GIL during inference, so several threads can have a batch in flight in one process. Surfra does not use Ray for this.