Services

cairn runs with none of these. Each is opt-in and needs the same three things: a Python extra, a compose profile (if it runs as a container), and a few environment variables. This page lists all three per service.

Enable profiles with COMPOSE_PROFILES in deploy/prod/.env (comma separated) and matching --profile flags:

COMPOSE_PROFILES=temporal,mlflow,otel
docker compose --env-file .env --profile temporal --profile mlflow up -d

Validate before starting anything — it cross-checks the .env against the profiles you enabled:

cairn deploy doctor --env deploy/prod/.env --profile temporal --strict

Two .env gotchas

No inline comments on a value line, and escape $ in an argon2 hash as $$ (compose interpolates $). Both bite silently.

Temporal — durable runtime

Survives restarts, retries per activity, and holds human-approval waits for days. Without it, runs execute in-process and a restart loses them.

uv sync --extra runtime-temporal --extra runtime-postgres
docker compose --env-file .env --profile temporal up -d

Profile temporal starts three containers: the Temporal server (temporalio/auto-setup, backed by the same Postgres), its UI on 127.0.0.1:8233, and a cairn worker running the cairn-worker console script. Add dedicated-temporal-db to give Temporal its own Postgres.

Variable	Default	Purpose
OBS_RUNTIME	inprocess	set temporal to route runs to the cluster
OBS_TEMPORAL_ADDRESS	localhost:7233	cluster address (temporal:7233 in compose)
OBS_TEMPORAL_NAMESPACE	default	namespace
OBS_TEMPORAL_TASK_QUEUE	cairn	base queue; the worker build id is appended
OBS_WORKER_BUILD	installed version	build id stamped on the queue
OBS_TEMPORAL_REMOTE_TASK_QUEUE	unset	dedicated queue for remote/GPU legs
OBS_TEMPORAL_DATASET_TASK_QUEUE	unset	dedicated queue for dataset batches
OBS_TEMPORAL_ALLOW_EMPTY	unset	let the worker boot with zero servable graphs

The checkpointer must be durable too

Set OBS_CHECKPOINTER=postgres and OBS_CHECKPOINTER_DSN on the server and the worker. deploy doctor fails the combination of Temporal with the default in-memory checkpointer, because a restart would strand every paused approval.

The worker needs SSH_PRIVATE_KEY too if any leg runs on a remote box.

Metaflow — ML flow engine

uv sync --extra engines
docker compose --env-file .env --profile metaflow-service up -d

Profile metaflow-service starts the Netflix metadata service on 127.0.0.1:8080 plus its own Postgres.

Variable	Default	Purpose
OBS_ML_ENGINE	builtin	set metaflow to run kind: ml flows on it
OBS_DATAFLOW_ENGINE	builtin	same selector for kind: data flows
OBS_METAFLOW_SERVICE_URL	http://metaflow-service:8080	metadata service
OBS_METAFLOW_ROOT	~/.cairn/metaflow	local metadata root
OBS_METAFLOW_DATASTORE	local	s3 for shared artifacts
OBS_METAFLOW_DATASTORE_ROOT	—	required when the datastore is s3
OBS_METAFLOW_COMPUTE	local	local | kubernetes | batch
OBS_METAFLOW_IMAGE	—	required for kubernetes/batch
OBS_METAFLOW_QUEUE	—	AWS Batch queue name
OBS_METAFLOW_RETRY	—	substrate-level retry count
OBS_METAFLOW_SECRETS	—	comma-separated connection names to inject
OBS_METAFLOW_ISOLATION	auto	auto | inherit | strict

cairn derives the downstream METAFLOW_* variables itself — you set the OBS_* ones only.

Prefect — data flow engine

Prefect is not part of the compose stack; you point cairn at an existing Prefect deployment.

uv sync --extra engines
export OBS_DATAFLOW_ENGINE=prefect
export PREFECT_API_URL=http://127.0.0.1:4200/api    # required — unset is a hard error

The Prefect side must also run cairn’s bridge, registered as the deployment cairn-dataflow-bridge/cairn:

python -m operator_dataflow.prefect_bridge

RAG and vector stores

uv sync --extra rag --extra rag-sqlite-vec      # or rag-pgvector / rag-qdrant

RAG_VECTOR_STORE selects the backend (default sqlite_vec). An unrecognised value silently falls back to sqlite-vec with a warning, so check doctor if retrieval looks empty.

Backend	Extra	Configuration
sqlite_vec (default)	rag-sqlite-vec	OBS_SQLITE_VEC_DB (falls back to OBS_RUN_DB, then ~/.cairn/runs.db)
pgvector	rag-pgvector	PGVECTOR_DSN required (hard error if unset). The prod compose Postgres is already a pgvector image.
qdrant	rag-qdrant	QDRANT_URL (default http://localhost:6333), QDRANT_API_KEY. Runs as a container in the dev compose file (--profile qdrant); external in prod.
pinecone	—	Not implemented — every method raises. Use qdrant or pgvector.

Embedding and retrieval settings, whichever store you pick:

Variable	Default	Purpose
RAG_EMBED_PROVIDER	gemini	gemini | openai; unknown values fall back to gemini
GEMINI_API_KEY / OPENAI_API_KEY	—	key for the chosen embedder
RAG_EMBED_MODEL	gemini-embedding-001	the OpenAI embedder overrides to text-embedding-3-small
RAG_EMBED_DIM	3072	1536 for the OpenAI default
RAG_COLLECTION_NAME	obs_runbooks	collection/table name
RAG_RERANKER	identity	identity | cross_encoder | cohere
OBS_RAG_AUTOINDEX	off	index declared corpora at startup

Index pack-declared corpora on demand with cairn knowledge index-packs — declared-but-unindexed knowledge retrieves nothing.

LiteLLM — the LLM gateway

One proxy in front of every model: routing, spend tracking, and an admin UI.

docker compose --env-file .env --profile llm-gateway up -d

No Python extra — the gateway is pure HTTP configuration.

Variable	Purpose
LITELLM_MASTER_KEY	required by compose when the profile is on (it fails fast without it)
LLM_GATEWAY	direct (default) or proxy
LLM_GATEWAY_URL	e.g. http://litellm:4000. Empty with proxy warns and silently falls back to direct
LLM_GATEWAY_API_KEY	same value as LITELLM_MASTER_KEY
LLM_GATEWAY_JOB_TAG_HEADER	default x-job-tag, for per-job spend attribution
LLM_GATEWAY_ADD_PROVIDER_PREFIX	default true

Model routing itself lives in deploy/prod/litellm-config.yaml.

MLflow — experiment tracking and the model registry

docker compose --env-file .env --profile mlflow up -d

Set MLFLOW_TRACKING_URI (http://mlflow:5000 self-hosted, or any managed URL with no profile at all). Without it, training logs nothing to MLflow and model/register keeps only the local append-only registry.

Compose-only knobs: MLFLOW_BACKEND_STORE_URI (defaults to SQLite inside the volume — move it to Postgres for real use) and MLFLOW_PUBLIC_HOST (must include your hostname or MLflow rejects proxied requests). Behind basic auth, also set MLFLOW_TRACKING_USERNAME / MLFLOW_TRACKING_PASSWORD.

Note

In the shipped compose file MLFLOW_TRACKING_URI is passed to the worker only. Add it to the server block too if server-side code needs to reach MLflow.

OpenTelemetry

uv sync --extra telemetry
docker compose --env-file .env --profile otel up -d

Variable	Purpose
OTEL_EXPORTER_OTLP_ENDPOINT	e.g. http://otel-collector:4318 — set on both server and worker; presence is what enables tracing
OTEL_EXPORTER_OTLP_HEADERS	auth headers for a managed collector
OTEL_SERVICE_NAME	default cairn (the worker reports cairn-worker)
OTEL_CONSOLE	also print spans locally, for debugging
OBS_TRACE_CONTENT	off | redacted (default) | raw — controls prompt/completion export

The collector fans out to your backends (Langfuse, Sentry, and any OTLP endpoint) via deploy/otel-collector.yaml, reading LANGFUSE_OTEL_ENDPOINT / LANGFUSE_OTEL_AUTH, SENTRY_OTEL_ENDPOINT / SENTRY_OTEL_AUTH, and OTEL_BACKEND_ENDPOINT / OTEL_BACKEND_AUTH from the same .env. Add --profile observability for Prometheus plus Grafana (127.0.0.1:3001).

All profiles

Profile	Starts
(none — always on)	caddy, server, postgres
temporal	Temporal server, Temporal UI, cairn worker
dedicated-temporal-db	a separate Postgres for Temporal
metaflow-service	Metaflow metadata service + its Postgres
llm-gateway	LiteLLM proxy
mlflow	MLflow tracking server
otel	OpenTelemetry collector
observability	Prometheus + Grafana
nats	JetStream-backed durable trigger inbox
infisical	self-hosted secrets (+ Postgres, Redis)
backup	nightly pg_dump

Known wrinkle

deploy doctor doesn’t know about metaflow-service or llm-gateway, so it prints an “unknown profile” warning for them. Both are valid; the warning is cosmetic.

Required regardless

Compose fails fast without CADDY_DOMAIN, POSTGRES_PASSWORD, OBS_JWT_SECRET, and OBS_API_PASSWORD_HASH (plus LITELLM_MASTER_KEY when the gateway profile is on). deploy/prod/.env.example documents every setting, and cairn doctor shows what the running process actually resolved.