Pipeline catalog

Every pipeline below ships with cairn and is public: it appears on the console’s Pipelines page unless an admin hides it. (cairn pipeline ls lists every installed pipeline, internal ones included.) Inspect one before you run it:

cairn pipeline show pipeline:training/finetune      # inputs, legs and gates
cairn pipeline explain pipeline:training/finetune   # what each leg will do, the gates, the budget

Legs counts the pipeline’s top-level legs. Gates counts data gates and person gates (see Gates and approvals). Budget is the series budget in governance (see Cost and budgets). A schedule means enabling the pipeline arms it.

Fine-tune and train a model

Pipeline	What it does	Legs	Gates	Budget
`training/e2e-finetune`	Governed end-to-end fine-tune — dataset readiness, QLoRA training, a human promotion gate, then a cited report.	3	1 person	$300
`training/facts-window`	Synthesize one post-cutoff fact window into closed-book QA and prep its train slice, registering both as governed dataset versions.	2	—	$10
`training/finetune`	Governed fine-tuning: compose the dataset-readiness gate, price the run before a GPU is provisioned, train a QLoRA adapter, gate it on held-out metrics, then promote it through a durable human approval — narrated into a cited report.	1	—	$40
`training/proxy-rank`	Rank a portfolio of training configurations with cheap proxy runs on a deterministic subsample, prune with successive halving, and propose the winner for a full run.	1	—	$15
`training/retrain-cycle`	A bounded retrain cycle: take a window of fresh governed signal, mix prior-cycle rows back in, continue the previous cycle’s adapter, measure the new adapter against the one it would replace on the same harness, gate on both the improvement AND the absence of forgetting, and promote only through a human decision.	1	4 data, 2 person	$300
`training/verifiable-finetune`	Synthesize post-cutoff facts into closed-book QA, measure the base model (novelty + hallucination), fine-tune on the train facts, serve the adapter, re-measure on held-out facts / paraphrases / control, and gate promotion on the measured delta.	10	2 data, 2 person	$50

Prepare a dataset

Pipeline	What it does	Legs	Gates	Budget
`dataset_readiness/api-and-admit`	Page a JSON API, run the rows through admission (normalize, dedup, decontamination, PII redaction) and register the cleaned export as a named, hash-pinned dataset version — resumable via an explicit watermark.	3	1 person	$1
`dataset_readiness/clean`	Clean a dataset in one governed pass — normalize, drop duplicate rows, redact PII, export the result — with a report of exactly what changed.	1	—	$1
`dataset_readiness/decontaminate`	Measure overlap (exact, n-gram or MinHash) of a training set against a benchmark: how many rows are contaminated, which ones, and by how much.	1	—	—
`dataset_readiness/import`	Import an external dataset through admission checks (normalize, dedup, decontamination, PII redaction) and register the cleaned export as a named, hash-pinned dataset version.	1	—	$1
`dataset_readiness/prep`	Prepare a dataset for training: scrub PII, then write a deterministic, leakage-free train/val/test split with provenance and a post-scrub verification.	1	—	$1
`dataset_readiness/promote`	Promote a dataset version to the tag everything downstream reads, with the previous target recorded.	1	—	—
`dataset_readiness/pull-and-admit`	Pull new rows from a SQL source, run them through admission (normalize, dedup, decontamination, PII redaction) and register the cleaned export as a named, hash-pinned dataset version — resumable via an explicit watermark.	3	1 person	$1
`dataset_readiness/readiness`	Assess a corpus, remediate it, and re-assess — bounded, with a human decision when it will not converge. Runs on a schedule.	2	—	—
`dataset_readiness/watch-source`	Check a watched source (a registered dataset, an s3 prefix, or a table), admit what is new through the governed import as a hash-pinned dataset version, then propose the downstream run for approval — or start it, when the declaration explicitly opted in.	7	—	$1

Serve and run inference

Pipeline	What it does	Legs	Gates	Budget
`serving/batch-inference`	Run a prompt over every row of a dataset against a served model, with a pre-spend cost gate, a hard row ceiling, and the real token spend recorded against the series afterwards.	4	1 data	$25
`serving/export-model`	Export a trained adapter as merged 16-bit weights, a GGUF at a chosen quant, an Ollama Modelfile carrying the training chat template, and a vLLM-ready serving stanza.	1	—	—
`serving/serve`	Stand up a model endpoint and benchmark it before anyone is pointed at it.	2	1 person	—

Evaluate and compress

Pipeline	What it does	Legs	Gates	Budget
`eval/score-a-model`	Score a model against benchmark suites, and flag regressions against a baseline.	2	1 person	—
`compress/distill`	Distil a large teacher into a small student adapter and hold promotion at a human gate.	2	1 person	$30
`compress/quantize`	Quantize a model, then gate on the measured quality delta.	2	1 person	—

Estimate cost

Pipeline	What it does	Legs	Gates	Budget
`compute/price-a-run`	What a run would cost — live GPU offers beside cairn’s own estimate.	1	—	—

Operations and security

Pipeline	What it does	Legs	Gates	Budget
`incident/rca`	Investigate a production incident, then write the postmortem once a human has accepted the root-cause finding.	3	1 person	—
`security_triage/triage`	Triage a security alert into a severity and a recommended action.	1	—	—

Run your first pipeline
Write a pipeline: start from one of these.
Operators: the operators these pipelines call.