Run your first pipeline

This guide runs the public dataset_readiness/import pipeline: it cleans a JSONL file and registers the result as a dataset version. The same steps work for any pipeline in the catalog.

You need a running cairn: cairn dev from the Quickstart, or a deployed server.

1. Find it and see what it will do

In the console, Pipelines lists every pipeline enabled in your workspace, grouped by pack. Each card shows its legs and gates. In the launch dialog, Show what will run and Preflight describe the plan without running it.

From the terminal, cairn pipeline ls lists installed pipelines and cairn pipeline show prints one pipeline’s legs and gates:

$ cairn pipeline ls
  compress/distill  2 legs (gate,task)  sha256:c7abf33956b31818
  …
  dataset_readiness/import  1 legs (flow)  sha256:d2698829f06eef29
  …
  serving/batch-inference  4 legs (flow,gate,task)  sha256:54747ab7ecffb057

cairn pipeline explain goes further. For each leg it shows the file that will run, the engine, every gate on the way and why it applies, each input and where its value comes from. It runs nothing and spends nothing:

$ cairn pipeline explain dataset_readiness/import -e sandbox
pipeline:dataset_readiness/import  —  Import an external dataset through admission checks …
  environment: sandbox    runtime=inprocess  storage=sqlite  bus=in-process
  governance:  {'budget': {'usd': 1}, 'series': 'dataset-import'}

  INPUTS (value ← layer)
    path                                                               ← required — UNSATISFIED
    eval_path                                                          ← default (def)
    register                                                           ← required — UNSATISFIED

  [flow] clean  flow:dataset_readiness/intake  (entrypoint)

2. Launch it

Console. On the pipeline’s card select Run. Fill the required inputs (path and register). Under Advanced · environment, saved inputs, set Environment to sandbox, or leave it blank for the workspace default. Select Launch. You can also save the inputs as a preset there.

CLI, on the machine that runs cairn. cairn pipeline run launches through the same door as the console and stays attached until the pipeline finishes:

cairn pipeline run dataset_readiness/import -e sandbox \
  -i path=/data/support-chats.jsonl \
  -i register=support-chats

Each -i key=value sets one input; values are parsed as JSON when they can be.

CLI, against a server. Sign in once, then launch remotely:

cairn remote login -e https://cairn.example.com -u alice
cairn remote launch dataset_readiness/import \
  -i path=s3://my-bucket/support-chats.jsonl \
  -i register=support-chats \
  --environment sandbox --watch

pipeline_id=pl_7ab851ad82184dfa status=running
  running
    clean              running              run=run-flow-dataset_readiness--intake-da9fcc
  done
    clean              done                 run=run-flow-dataset_readiness--intake-da9fcc
→ findings  cairn remote findings pl_7ab851ad82184dfa

3. Follow it, and answer its gates

The pipeline page shows each leg as it runs. If the pipeline reaches a gate that waits for a person, it pauses and the gate appears in Approvals, where anyone signed in to the workspace can read the evidence and decide. See Gates and approvals.

4. Read the outputs

On the pipeline page, Inputs & outputs and each leg’s run show what was produced. From the CLI, cairn remote findings prints every leg’s outputs:

$ cairn remote findings pl_7ab851ad82184dfa
pipeline pl_7ab851ad82184dfa  status=done
  clean  [done] run=run-flow-dataset_readiness--intake-da9fcc
    dataset_versions                          [1 items]   (--json for the full value)
    dedup_exact_dropped                       0
    expectations_check_verdict                pass
    ingest_normalize_rows                     2
    pii_redact_rows                           2

Next

• Write a pipeline: your own, with gates.
• Pipeline catalog: what else you can run.
• Runs and the record: everything the run left behind.