Jobs

A job runs one command, once, on one machine, and ends when the command exits. Use it for work that has a start and an end: a training run, a fine-tune, a dataset conversion, a nightly batch. For a model that answers requests, deploy an app instead.

	App	Job
Runs	A handler, for each request	One command
Machines	Scales from zero to many	Exactly one
Ends	When the autoscaler scales it down	When the command exits
Failure	The request is retried	The job fails. It is not retried

Image

The image needs the surfra package. Surfra starts the container with surfra job agent, which runs your command, ships its output, and reports how it ended. This replaces the image’s CMD. If the image has an ENTRYPOINT, it must run the command it is given, as exec "$@" does.

FROM python:3.12-slim
RUN pip install --no-cache-dir https://github.com/surfra-io/surfra-python-sdk/releases/download/v0.2.0/surfra-0.2.0-py3-none-any.whl
COPY train.sh /app/
WORKDIR /app

The command runs in the image’s WORKDIR through bash -c, or sh -c when the image has no bash, so pipes and && work.

Submit

From the command line, everything after -- is the command:

surfra job submit --image ghcr.io/acme/train:3 --runner desk-4090 -- bash train.sh --epochs 10

The command follows the job’s logs until it ends, then exits with the job’s exit code. Press Ctrl+C to stop following. The job keeps running. --no-wait prints the job id and returns right away.

From Python:

import surfra

client = surfra.SurfraClient()
job = client.submit_job(
    "bash train.sh",
    args=["--epochs", "10"],
    image="ghcr.io/acme/train:3",
    providers=[{"provider": "runner", "runner": "desk-4090"}],
)
for line in job.logs(follow=True):
    print(line["m"])
print(job.wait()["status"])

The dashboard’s Jobs page has the same form under New job.

Pick a machine

List machines in the order to try. The first one that can take the job runs it. A job that fits nowhere stays queued and its logs say why.

surfra job submit --image ghcr.io/acme/train:3 \
  --runner desk-4090 \
  --provider runpod --gpu a100 \
  -- python train.py

--runner names one of your machines and can be repeated. --provider with --gpu and --gpu-count picks a connected cloud. --providers-json takes the same rows an app’s autoscale uses, for anything else.

A runner runs one thing at a time. While a job has it, apps that list it skip to their next row.

Spot machines

--spot asks for a spot machine. It is cheaper, and the provider can take it back at any time. Surfra warns when you submit, and a job that loses its machine ends as lost. It is not restarted. Save checkpoints somewhere outside the machine if the run is long.

Environment

--env NAME=VALUE and --env-file set environment variables. --secret adds every value from a stored credential. --docker-secret pulls a private image.

Values are encrypted when they are stored. The job page and surfra job status show each variable’s name and where it came from, never its value.

When the same name is set twice, the later layer wins:

1. The --secret credential
2. The job spec’s env
3. The env given at submit

Surfra also sets these. Names starting with SURFRA_ are reserved.

Variable	Value
SURFRA_JOB_ID	The job_ id
SURFRA_JOB_NAME	The job’s name
SURFRA_API_KEY	A key for calling apps, only when --call-app is set

Call apps from a job

A job can call your apps, for example to embed a dataset with a deployed model. List the apps it may call:

surfra job submit --spec embed-corpus --call-app embedder --call-app reranker

The job gets a SURFRA_API_KEY that can only call those apps. It cannot deploy, read other jobs, or manage the account, and it stops working when the job ends. --call-app '*' allows every app. --deny-app blocks an app even when * is set.

Inside the job, the SDK reads the key from the environment:

import surfra

for vector in surfra.SurfraClient().map("embedder", ({"text": t} for t in texts), concurrency=16):
    ...

map calls the app once per item, keeps up to concurrency calls waiting at a time, and yields the outputs in order.

Without --call-app, the job gets no key.

Time limits

Flag	Default	What happens
--deadline	None	The job is stopped when it has run this long
--liveness	10m	When Surfra hears nothing from the job for this long, the job ends as lost and the machine is released. At least 1m
--grace	30s	Time between SIGTERM and SIGKILL when the job is stopped

Durations take 90, 90s, 10m, 3h, or 2d.

The agent sends a heartbeat every 15 seconds. If it loses contact with Surfra, the command keeps running and the agent keeps retrying. If contact comes back within the liveness window, nothing is lost. If it does not, the job is marked lost and the machine is released so it stops costing money. A job that has not started within 30 minutes of getting a machine fails.

Stop

surfra job stop job_...

The command gets SIGTERM, then SIGKILL after the grace period. Catch SIGTERM to save a checkpoint. A stopped job ends as stopped.

How a job ends

Status	Meaning
queued	Waiting for a machine
starting	A machine has it and the image is starting
running	The command is running
succeeded	The command exited with code 0
failed	The command exited with another code, or the job could not start
stopped	You stopped it, or it reached its deadline
lost	Surfra stopped hearing from it, or the machine went away

Jobs are never retried. To run it again, submit it again, or click Run again on the job page.

Logs

Standard output and standard error are stored with a timestamp. Lines from Surfra itself, such as where the job was placed and why it ended, are marked [surfra].

surfra job logs job_...
surfra job logs -f job_...

The job page shows the same lines, filters them by stream, and follows them live. Job logs are kept for 7 days.

Specs

A spec saves the image, command, machines, env, and limits under a name. Submitting with --spec fills in everything you leave out. Words after -- are added to the spec’s arguments.

surfra job spec save train-llm --image ghcr.io/acme/train:3 --runner desk-4090 \
  --env-file train.env --deadline 12h -- python train.py
surfra job submit --spec train-llm -- --lr 3e-4

client.save_job_spec("train-llm", image="ghcr.io/acme/train:3", entrypoint="python train.py",
                     providers=[{"provider": "runner", "runner": "desk-4090"}], deadline_sec=43200)
job = client.submit_job(spec="train-llm", args=["--lr", "3e-4"])

Fields given at submit win over the spec. Env from both is merged.

Retry a submit safely

Pass a submission id when a script might submit twice, for example after a timeout:

surfra job submit --spec train-llm --submission-id nightly-2026-10-15

Submitting again with the same id returns the job that already exists.

Billing

A job is billed for its machine time like an app. Surfra will not start a job when the balance cannot cover it. Once a job is running, it is never stopped for money. If the balance runs out, the job’s logs say so and the job keeps running.

On your own machine

A job on a runner runs as a Docker container labeled with the job id. If the runner restarts, it picks the container back up instead of starting the command again.

surfra runner stop refuses while a job is running, so a long run is not cut off by accident. surfra runner stop --force stops the job too.