Pipelines & bundles

A bundle chains N workflows — possibly from different packs — into a pipeline. Each step is a full governed run (its own policy gates, budget, provenance, audit); the pipeline layer adds deterministic hand-off between them: data wiring, gate predicates, triggers, and approvals.

The bundle file (bundle.toml)

[bundle]
name = "dataset-clean-finetune"
version = "0.1.0"
profile = "local-dev"

[[bundle.pipeline]]
name = "clean-then-finetune"
mode = "pipeline"                      # pipeline (default) | graph (legacy flatten)

[[bundle.pipeline.step]]
id = "check"
pack = "dataset-readiness"
template = "check"
inputs = { path = "$input.dataset_path" }

[[bundle.pipeline.step]]
id = "prep"
pack = "dataset-readiness"
template = "prep"
gate = "$steps.check.verdict != 'fail'"
inputs = { dataset_path = "$input.dataset_path" }

[[bundle.pipeline.step]]
id = "finetune"
pack = "training"
template = "finetune"
gate = "$steps.prep.post_scrub_pii_verified == true"
inputs = { dataset_path = "$steps.prep.anonymized_path" }

Step fields

Field	Meaning
id, pack, template	the step’s identity and the workflow it runs
inputs	data wiring: $input.<field> (bundle input) or $steps.<id>.<field> (an upstream step’s output)
gate	a declarative predicate over upstream output — == / != / bare truthiness. Never an LLM decision
trigger	auto (default) · manual · schedule:<cron> · on:<event>
approval	true = human approval between stages before this step starts
emits	a domain event published when the step completes (feeds on:<event> steps)
depends_on	explicit DAG edges; omitted = implicit linear order
environment	per-step environment/profile override

$steps.<id>.<field> reads a top-level field of the upstream step’s output (with a fallback into its findings.<field>) — no deep dotted paths. cairn bundle validate checks the whole data contract against the participating templates’ schemas at authoring time, and rejects wires that reference steps outside a step’s dependencies.

Running a pipeline

cairn bundle validate examples/bundles/dataset-clean-finetune
cairn bundle run examples/bundles/dataset-clean-finetune --input request.json

Over HTTP: POST /pipelines/run with {bundle_path, pipeline_name, inputs, environment} returns a pipeline_id; then GET /pipelines/{id}, POST .../steps/{step}/approve, POST .../cancel. The console’s Pipelines view shows each pipeline’s steps, gates, and paused approvals.

How stepping works

The pipeline bridge listens to the run-terminal event firehose: when a step’s run finishes, it evaluates the downstream gate against the persisted output, resolves the input wiring, and dispatches the next step as its own governed run. schedule: steps arm a cron timer; on:<event> steps subscribe to the bus — and on a durable bus (OBS_BUS=postgres) missed events are replayed from the moment the step started waiting, so a restart doesn’t drop the hand-off. On the default in-process bus, on: steps work within one process only (the server logs a warning).

On restart the bridge re-adopts in-flight pipelines: running steps re-map by run id, schedule and event waiters re-arm.

Durable pipelines on Temporal

With OBS_RUNTIME=temporal, pipelines run as a durable cairn.PipelineWorkflow: Temporal’s event history owns the orchestration, gates are re-evaluated in activities before every submit, schedule: steps sleep on deterministic timers, and manual/approval steps wait on signals. Bus events are forwarded to the workflow as signals. Run a worker with cairn worker.