Flows

A flow is one unit of work: clean a dataset, train an adapter, investigate an incident. A flow is a component. It runs only as a leg of a pipeline.

Three kinds

Kind	File	What it is	Where it runs
data	flows/<name>.yaml	an ordered list of stages over a dataset	a dataflow engine (built-in, Prefect, Metaflow, SkyPilot)
ml	flows/<name>.yaml	the same shape, for training and evaluation	the ML engine configured for the environment
agentic	flows/<name>.md	a markdown file: front matter plus a prompt or a state machine of nodes	cairn’s LangGraph runtime

A pipeline leg runs a data or ML flow with flow: and an agentic flow with template:. cairn flow ls lists all three kinds:

  compress/closedbook-eval  kind=data stages=2
  dataset-readiness/intake  kind=data stages=8
  serving/batch-infer  kind=data stages=3
  training/dpo-align  kind=ml stages=1

What a flow takes and produces

A data or ML flow reads its settings from $config.<key> placeholders in its stage list. The leg’s with: block fills them. A key the leg leaves out is dropped, so the stage’s own default applies.

name: readiness
kind: data
stages:
  - stage: ingest.normalize
    config:
      path: $config.raw
  - stage: dedup.exact
  - stage: pii.redact
    config:
      backend: $config.pii_backend
  - stage: export.jsonl

When it finishes, the leg’s output has export (the file the last stage wrote) and findings: one number or value per stage, named <stage>_<field>, for example pii_redact_rows or dedup_exact_dropped. Later legs and gates read them.

An agentic flow declares typed input_schema and output_schema in its front matter. Its nodes call operators and write their results into the flow’s findings.

A pipeline wraps a flow

To run a flow by itself, give it a one-leg pipeline. $inherit copies the leg’s inputs into the pipeline, so you do not repeat them:

name: import
kind: pipeline
inputs:
  $inherit: clean
  path: {type: Dataset, description: "Staged upload path or s3:// uri of the file to import."}
  register: {type: str, description: "Dataset-registry name to version the cleaned export under."}
steps:
  - id: clean
    flow: intake
    produces: {export: Dataset}

That is abridged from the public dataset_readiness/import pipeline, which wraps the intake flow.

Starting a flow directly is refused, and the message names the fix:

'acme/scrub' is a flow (data), and a flow does not run on its own (ADR-0106 D1):
only a declared pipeline runs, so a flow with no wrapper is a component.

The dev loop

While you write an agentic flow, cairn run can run it in your terminal, in this process, without a pipeline:

cairn run acme/hello --input input.json --dry-run       # the plan; nothing runs
cairn run acme/hello --input input.json --mock          # operators take their mock path
cairn run acme/hello --input input.json --interactive   # approve or reject gates at the prompt

These runs write no pipeline record and no run row. --mock and --interactive write a local trace instead, which cairn replay <run-id> prints. --mock replaces operators only: a prompt flow’s model call is still made, and needs a key.

The dev loop resolves agentic flows only. Try a data or ML flow through a wrapper pipeline in the sandbox, as Write a flow shows.

Next

• Capabilities and environments: which operator does the work.
• Write a flow: a small data flow from scratch.
• Pipeline and flow YAML: every field.