Runtimes & durability

How durable a run is, is a deployment choice, not a code change. Two orthogonal knobs:

• OBS_RUNTIME — who executes the compiled graph.
• OBS_CHECKPOINTER — where paused state (HITL gates, multi-turn ask threads) lives.

Runtime tiers

OBS_RUNTIME	What you get	Needs
inprocess (default)	the graph runs in the server process; no crash recovery	nothing
dbos	single-node durable: each run is a DBOS workflow; pending runs recover on restart; durable HITL. Library only — no extra server	runtime-dbos extra
temporal	each run is a Temporal workflow: per-node retries, durable replay, dedicated workers, a separate remote/GPU task queue	runtime-temporal extra + a Temporal cluster (the compose temporal profile ships one)

Selection is read per run, so switching tiers is an env change + restart. Note one temporal limitation: agent_loop nodes aren’t supported on the Temporal tier (prompt and state-machine flows are); they fail loudly rather than silently degrading.

Checkpointers

OBS_CHECKPOINTER	Durability of a paused run
memory (default)	resume works within the process; lost on restart
sqlite	survives restarts on one node (OBS_CHECKPOINTER_PATH)
postgres	survives restarts across processes (OBS_CHECKPOINTER_DSN)

A run paused at an approval gate is parked exactly at its interrupt() point; resume recompiles the template and continues from the checkpoint under the run’s thread_id. The same checkpointer backs durable ask conversations.

Workers

On the Temporal tier the server submits workflows; workers execute them:

cairn worker                 # main task queue
cairn worker --remote        # the dedicated remote/GPU queue

A worker registers both the template workflow and the pipeline workflow. Nodes labeled for remote execution route their activities to the remote queue — that’s how a GPU box participates: it runs cairn worker --remote and pulls only GPU-bound work.

Remote jobs

Long-running off-box work (GPU training, vLLM serving) runs through the remote-job control loop: the operator submits the job, the run pauses on a remote_job interrupt (status stays running — it’s machine-driven, not a human gate), and the control loop polls/streams progress and resumes the graph when the job completes. Jobs support cancel, and are visible in the console’s Jobs view and GET /console/api/jobs.