Pipeline and flow YAML

Both file kinds live in a pack: pipelines in <pack>/pipelines/<name>.yaml, data and ml flows in <pack>/flows/<name>.yaml. Agentic flows are markdown templates and aren’t covered here. Unknown or misspelt fields are errors at load, not silently ignored.

Pipeline file

name: batch-inference
kind: pipeline
description: Run a prompt over every row of a dataset, with a pre-spend cost gate.

inputs:
  model: {type: string, description: "Model id to call."}
  max_cost_usd: {type: float, default: 5, description: "A job predicted above this does not start."}

governance:
  budget: {usd: 25}
  series: batch-inference
  approvers: platform

steps:
  - id: estimate
    task: cost/tokens
    with:
      model: $input.model
  - gate: cost
    depends_on: [estimate]
    predicate: steps.estimate.usd <= input.max_cost_usd
  - id: infer
    flow: batch-infer
    depends_on: [cost]

That example is abridged from the shipped serving/batch-inference.

Top-level fields

Field	Type	Meaning
name	string, required	The pipeline’s name. Its ref is <pack>/<name>.
kind	pipeline	Marks the file as a pipeline.
description	string	Shown in the catalog and in cairn pipeline show.
visibility	public · internal · deprecated	Default public. Internal pipelines don’t appear on the Pipelines page.
inputs	map	The launch inputs: {name: {type, default, description}}. See below.
outputs	map	Names this pipeline’s results: {name: <step>.<field>}.
governance	map	budget: {usd: N}, the spending cap for the series; series, the series name the budget belongs to; approvers, who is asked at person gates that don’t name their own. See Cost and budgets.
triggers	list	How the pipeline starts without you. See Triggers.
outcome	string	What a person achieves with it, in their words. The catalog groups on it.
suite	string	The catalog bucket the outcome belongs to (for example model, data, operate).
compare	{before: <leg>, after: <leg>}	Two legs the before/after view shows side by side, for example base model against tuned model.
steps	list, required	The legs. See below.

Inputs

Each input is {type, default, description}. An input without a default is required at launch, and cairn pipeline run refuses to start without it.

Two reserved keys save repeating a flow’s own contract:

inputs:
  $inherit: run          # adopt the inputs of the leg with id `run`
  $hide: [mock_metrics]  # and drop these

Steps

Every step is exactly one kind, named by the key that carries its target:

Key	The leg
flow: <name>	runs a data or ml flow from the same pack
template: <name>	runs an agentic flow (a markdown template) from the same pack
task: <operator>	calls one operator, for example cost/tokens
gate: <name>	decides whether what follows may run; does no work
loop: <name>	repeats its own steps: until a condition holds

A leg can only reference flows and templates in its own pack; a reference to another pack is a load error.

Fields any step can carry:

Field	Meaning
id	The step’s id. Gates and loops are named by their gate: / loop: value instead.
with	Inputs passed to the leg. $input.<name> reads a pipeline input; $steps.<id>.<field> reads an earlier leg’s output.
depends_on	Steps that must finish first.
when	An expression. False skips this leg (recorded as routed off) and the pipeline continues.
label	A human-readable name for the run page and the graph. Display only.
environment	Run this leg in a named environment instead of the pipeline’s.
timeout	A limit for this leg: seconds, or a number with d/h/m/s. A value that doesn’t parse is a load error.
finalizer	true runs this leg once when the pipeline ends, whether it completed, failed or was cancelled, for example to release a machine. It must be safe to run twice.
consumes / produces	Name the artifacts the leg reads and writes, for wiring and lineage.

Gate fields (see Gates and approvals):

Field	Meaning
predicate	An expression. True passes the gate at once; false stops the pipeline. A data gate.
approvers	Who is asked. Without it, governance.approvers applies. A person gate.
show	<step>.<field> values shown to the approver. Any other form is a load error.

predicate and approvers can’t both be set on one gate.

Loop fields:

Field	Meaning
steps	The legs repeated on each pass.
until	An expression; the loop ends when it is true.
max_iterations	Required. The most passes the loop may make.
on_exhausted	gate (default): when the passes run out, a person decides. fail: the pipeline fails.

Expressions

when, predicate and until are JMESPath expressions over the pipeline’s inputs and its steps’ outputs, checked when the file loads:

predicate: steps.estimate.usd <= input.max_cost_usd
when: input.prep_enabled
until: steps.score.findings.passed

Write a literal number in backticks: input.rows > `1000`.

Triggers

A pipeline can start on its own. Each entry has a kind:

Kind	Starts the pipeline	Main fields
manual	only when someone launches it (the default)	description
schedule	on a cron schedule	cron; inputs; environment; misfire (skip, fire_once or fire_all); enabled (default false: enabling the pipeline arms it)
webhook	when a matching request arrives	source; path; match; input (maps request fields to inputs); enabled
watch	when a watched source changes or a signal crosses a threshold	source (kind: dataset, s3 or table); every; when; on_change (propose, start or none); inputs

triggers:
  - kind: schedule
    cron: "0 3 * * *"
    misfire: skip

Data and ml flow file

name: batch-infer
kind: data
description: Run one completion per row against a served model and export the rows.

stages:
  - stage: ingest.normalize
    config:
      path: $config.path
  - stage: infer.batch
    config:
      model: $config.model
      max_rows: $config.max_rows
  - stage: export.jsonl
    config:
      register: $config.register   # optional: version the export as a dataset

Abridged from the shipped serving/flows/batch-infer.yaml.

Field	Meaning
name	The flow’s name. A pipeline references it as flow: <name>.
kind	data (produces a dataset) or ml (produces a model). The flow never names an engine; the environment maps each kind to one.
description	Shown in cairn flow ls and the console.
stages	Required, at least one. Run in order over one working dataset.
handoff	{export: <stage>, ingest: <stage>}. Needed only when a stage declares gate:, so the working dataset can cross the pause.
checks	{<check>: not_applicable} declares that a known check, such as decontam, doesn’t apply to what this flow produces. It is recorded on the leg’s evidence.

Each stage:

Field	Meaning
stage	The stage’s registered name, for example dedup.exact.
config	The stage’s settings. $config.<name> reads the value the pipeline passed in with:.
resources	What the stage needs to run, for engines that schedule resources.
checkpoint	true saves the working set after this stage, so a restart resumes from here. Off by default.
gate	An approval required before this stage runs.
map	Fans the stage out over a list at run time: over (the list), as (the item name, default item), max_parallel (default 4, at most 64) and merge.

Next

• Write a pipeline
• Write a flow
• CLI