Pipelines, legs and steps

A pipeline is the unit you launch. It is a YAML file that names the legs to run, the order they run in, and the gates between them. Only a declared pipeline runs: there is no other way to start work in cairn.

Legs

A pipeline is a list of legs. Each leg has an id and exactly one kind.

Kind	What the leg does	Has its own run
flow:	runs a data or ML flow on a dataflow engine	yes
template:	runs an agentic flow (a markdown flow)	yes
task:	calls one operator directly	yes
gate:	stops until a predicate passes or a person approves	no
loop:	repeats a group of legs until a condition holds, with a hard limit	no

A leg that does work (flow, template, task) becomes its own run: its own status, cost and record, launched through the same admission checks as any other run. A gate has no run; its verdict and approver are stored on the pipeline. Gates and approvals covers them in full.

Legs run in the order of depends_on. A leg with no depends_on waits for the leg written before it. A leg with a when: condition that is false is skipped and the pipeline carries on.

Inside a flow, the units of work are steps: the stages of a data flow, or the nodes of an agentic flow.

An example

The public serving/batch-inference pipeline prices a batch job, stops if the price is too high, runs the job, then records what it really cost:

flowchart LR
  estimate["estimate<br/>task"] --> cost{"cost<br/>gate"}
  cost -->|"price ≤ cap"| infer["infer<br/>flow"]
  infer --> spend["spend<br/>task"]

Abridged from cairn/packs/serving/pipelines/batch-inference.yaml:

steps:
  - id: estimate
    task: cost/tokens
    with:
      model: $input.model
      rows: $input.rows

  - gate: cost
    depends_on: [estimate]
    predicate: steps.estimate.usd <= input.max_cost_usd

  - id: infer
    flow: batch-infer
    depends_on: [cost]
    with:
      path: $input.path
      max_rows: $input.rows

  - id: spend
    task: cost/tokens
    depends_on: [infer]
    with:
      model: $input.model
      measured: true
      rows: $steps.infer.infer_batch_rows

Values are wired with expressions. $input.<name> reads a pipeline input and $steps.<leg>.<field> reads an earlier leg’s output. The whole value must be the expression; there is no string interpolation. The YAML reference lists every field.

The launch door

The console’s Run button, cairn pipeline run, cairn remote launch and a schedule all start a pipeline through the same launch code. In order, it:

1. resolves the reference (<pack>/<name>, optionally prefixed pipeline:),
2. checks the inputs and coerces them to their declared types,
3. applies policy: a policy can insert an approval gate before a leg, or refuse the launch outright,
4. runs admission checks (for example, a policy can require that the dataset version a pipeline trains on passed named checks),
5. only then writes the pipeline record and starts the first legs.

A launch refused in steps 1 to 4 leaves no record behind.

A pipeline record moves through queued, running, paused (something is waiting for a person), and ends done, failed or cancelled.

Triggers belong to pipelines

A pipeline can declare a schedule. Flows cannot: a flow only runs as part of a pipeline.

triggers:
  - kind: schedule
    cron: "0 3 * * *"
    inputs:
      path: s3://corpus/current
    enabled: false

A schedule is armed when the pipeline is enabled in a workspace. It starts disabled unless you set enabled: true, so enabling a pipeline never starts unattended spend by surprise. A fired schedule goes through the same launch door as a person pressing Run.

Next

• Flows: the components legs run.
• Gates and approvals: data gates, person gates and policy gates.
• Write a pipeline: build one step by step.