ModelOps lifecycle

The flagship use of the engine is the governed model lifecycle. Every stage is a normal governed run; chained together (by the training/finetune template’s subgraphs, or as an explicit pipeline like examples/bundles/dataset-clean-finetune) they form one auditable spine.

1. Gate the dataset

dataset-readiness/check runs six deterministic checks — format, duplicates, train/eval contamination, PII density, distribution profile, provenance — through dataset/gate into a pass/warn/fail verdict, then narrates a cited readiness report. data/profile validates that the rows actually fill the workflow’s declared data roles before anything runs.

Honesty note: dataset/pii_scan samples 1,000 rows by default; set sample_size: null for a full scan, and prefer the Presidio backend (the production profile pins it).

2. Prep it

dataset-readiness/prep scrubs PII (dataset/anonymize), splits train/val/held-out-test (dataset/split), re-verifies the scrub, and passes a deterministic prep_gate; a dirty scrub can route through dataset/prep_review — a durable human review gate.

3. Train

training/finetune composes the readiness check as a subgraph, gates on it, estimates cost (compute/estimate + cost/gate — spend verdict before a GPU is provisioned), then trains:

• train/local — real LoRA SFT on CPU/MPS for small jobs;
• remote/exec — the SSH path: the dataset is staged to S3 (train/stage-dataset) and a real GPU box runs the training script as a cancellable remote job;
• train/modal / train/runpod / train/vast / train/kaggle — provider APIs.

The zero-setup default binding is train/stub — it fabricates a plausible result so the workflow is try-able without spend. Production deployments gate this out: the production profile’s model/gate sets allow_stub=false.

4. Evaluate — deterministic gates, advisory judges

The eval layer separates hard signals from advisory ones:

• model/gate — the deterministic pass/warn/fail from training metrics; in the production profile it also require_benchmark=true and blocks on regression.
• eval/benchmark — IFEval/GSM8K on base vs adapter. Its output carries checked — a stub score with checked=false is what you get without the extra or a GPU, and a fail-closed gate rejects it.
• eval/delta — vs the last registered baseline (model/baseline) under an explicit regression budget.
• eval/judge — an LLM rubric judge, off by default and never gating.

5. Promote behind a human

model/register is requires_approval — the run parks as awaiting_approval on a durable gate; a human approves in the console, CLI, or Slack, and the resume is recorded (who, when, channel) in the hash-chained audit log. Registration lands in MLflow when MLFLOW_TRACKING_URI is set (the mlflow compose profile ships one), else a local registry. model/card / report/model_card render the auditor-facing card, and lineage/record writes the content-addressed lineage that cairn lineage <model> later verifies chain-intact.

6. Serve and load-test

serving/serve deploys base + adapter behind an OpenAI-compatible vLLM endpoint on an SSH GPU box (serve/vllm), then measures it for real (infer/bench: latency percentiles, throughput) into a ServeReport.

7. Compress (optional)

compress/quantize re-runs the eval after quantization and gates the delta against a regression budget before an HITL promote — same spine, smaller model.

Watching it run

Every stage streams live (SSE / cairn remote logs -f / the console), costs accrue on the run ledger against the budget cap, and the whole chain is replayable: cairn replay per run, cairn lineage per model, the Audit view for the governance receipt.