Batch

With a batch size above 1, the handler receives a list and returns a list of the same length, in the same order.

app.py

import surfra
from transformers import pipeline

generator = None

@surfra.warmup
def load():
    global generator
    generator = pipeline("text-generation", model="Qwen/Qwen3-0.6B", device=0)

@surfra.handler(batch_size=surfra.batch_size(1, 8), batch_wait=300)
def generate(requests: list[surfra.Request]):
    prompts = [r.input["prompt"] for r in requests]
    outputs = generator(prompts, max_new_tokens=64, batch_size=len(prompts))
    return [{"text": out[0]["generated_text"]} for out in outputs]

A worker takes up to 8 queued requests. If fewer are waiting, it waits up to 300 ms for more before it starts.

deploy.py

import surfra

app = surfra.App("qwen-batch")
app.deploy(
    image="ghcr.io/you/qwen-batch:1",
    entrypoint="app.py",
    runtime={"engine": "python"},
    autoscale={
        "max_replicas": 4,
        "providers": [
            {"provider": "salad", "gpu": "rtx3090", "priority": 1},
            {"provider": "runpod", "gpu": "rtx4090", "priority": 2},
        ],
    },
)

calls = [app.enqueue({"prompt": f"Write a haiku about {topic}"}) for topic in ["rain", "tea", "trains"]]
for call in calls:
    print(call.result(timeout=900))

The image needs transformers and torch on top of the Python Dockerfile. See Batching and concurrency for concurrent_workers and worker modes.