Autoscaling

Surfra decides how many replicas an app needs whenever a request arrives or finishes, and every 20 seconds in between.

Scaling up

Signal	Replicas wanted
min_replicas	Always at least this many
Requests queued or running	One per replica’s worth of them: queued plus running, divided by what one replica holds, rounded up
A workflow step is waiting on a step before it	At least one, so the next app is warm when the input arrives
http_ports with requests in flight	One per max_concurrent requests in flight

What one replica holds is max_concurrent on the HTTP engine, and the row’s num_workers on the Python engine. Both default to 1. Twelve queued requests with max_concurrent: 4 want three replicas.

The result is capped at max_replicas. Replicas that are still starting count toward it, so a burst never starts more machines than it needs. Each new replica walks the priority list and takes the first row that can place it.

Scaling down

A replica stops when all of these are true:

• The app has more replicas than it wants.
• The replica has no open requests.
• Its own last request finished at least idle_timeout_sec ago. A request served by another replica does not count. For a new replica, the clock starts when its worker registers.

idle_timeout_sec: 0 stops a surplus replica on the first check after its last request. A replica that is still starting is never stopped for being idle.

Surplus replicas stop from the bottom of the priority list first, so the machines you ranked highest stay. A replica whose workers all stopped heartbeating, because the machine was preempted or the process crashed, is stopped and replaced.

Failed boots

A replica that never registers a worker within boot_timeout_sec is stopped. One that reports a boot error, such as the HTTP server exiting or never answering its ready path, is stopped on the next check. The error stays on the replica in Workers and Logs.

Failed boots in a row slow down new starts:

Failed boots in a row	What happens
1–2	Replaced right away
3	No new replica for 1 minute
4 and up	The wait doubles each time, up to 30 minutes
5 and up	Queued requests fail with the boot error instead of waiting

The count resets when any replica of the app registers a worker. The app page shows the count and when the next start is allowed.

autoscale={
    "min_replicas": 1,          # keep one warm
    "max_replicas": 8,
    "idle_timeout_sec": 120,    # stop extras two minutes after their last request
    "providers": [...],
}

min_replicas: 0 scales to zero. The first request after that waits for a cold start: the image pull plus warmup.

Per-provider limits

Turn on per_provider to give each priority row its own minimum and maximum. This lets you keep cheap capacity running on one provider and send bursts to a faster or more available one.

autoscale={
    "max_replicas": 6,
    "per_provider": True,
    "providers": [
        # always keep one cheap replica on Salad, and never more than one there
        {"provider": "salad", "gpu": "rtx3090", "priority": 1, "min_replicas": 1, "max_replicas": 1},
        # everything above that goes to RunPod
        {"provider": "runpod", "gpu": "rtx3090", "priority": 2},
    ],
}

In the dashboard, switch on Per-provider scaling above the priority list. Each row gets Min and Max. Leave Max empty for no limit. The side panel shows the always-on cost and the burst rate for each row.

How placement works with limits on:

1. Minimums first. Each row keeps its min_replicas running, whether or not there is traffic.
2. Bursts in order. Extra replicas fill rows from the top, skipping any row at its max_replicas.
3. Fallback. If a row cannot place its minimum, for example Salad is out of RTX 3090s, the next row with room runs a stand-in. When the row can place again, Surfra starts the replica there and the stand-in stops once it is idle.
4. Scale-down. Surplus replicas stop from the bottom of the list, and no row drops below its minimum.

Rules:

• The app’s min_replicas becomes at least the sum of the row minimums.
• The row minimums must add up to no more than max_replicas.
• A row’s max_replicas must be at least its min_replicas.

A save that breaks a rule is rejected with a 400 that names the problem.

With per_provider off, row limits are ignored and every row is a fallback for the one above it.

Timeouts

Field	Meaning	Default
idle_timeout_sec	How long a surplus replica stays up after its last request. 0 stops it as soon as it is idle	300
boot_timeout_sec	How long a new replica has to register a worker	900, or the HTTP ready_timeout plus 600 if that is larger
execution_timeout_sec	How long one request may run on the HTTP engine, and how long a gateway call waits	1200 in the dashboard

Cost

You pay for replica seconds, not requests. A replica that is starting, idle, or busy is billed at its hourly rate. To lower cost, lower min_replicas and idle_timeout_sec. To lower latency, raise them.