mimir_analytics

Copy Markdown View Source

Analytical run-record store for the mimir agent stack: the canonical, queryable record of an agent run — workflow → step → run (session) → turn → tool_call, plus model calls, routing decisions, eval outcomes, and raw events — MotherDuck-hosted DuckLake in deployment, a local DuckDB file in dev/test, fed by idempotent JSONL-buffered ingest.

The correlation-id contract

Every mapper populates these columns, with exactly these names, whenever the source carries them: workflow_id, step_id, parent_step_id, run_id, mimir_request_id, decision_id, grant_id, virtual_key_id, parent_key_id, tenant_id, agent_digest. A consumer joins any two tables through the workflow/step/run spine without knowing which subsystem captured the row. This contract — not the table list — is the stable part; tables grow columns additively only.

Why mimir_request_id, not request_id

Every other family repo calls this column request_id. Here it's mimir_request_id on purpose: this store joins rows pulled from multiple sources (gateway request logs, session drops, eval reports), and several of those sources carry their own unrelated request_id-shaped column (e.g. the gateway export's upstream scope_id/request_id pair). Namespacing the mimir-side id avoids silent collisions/shadowing at mapper and query time.

Dependency direction (invariant)

mimir_analytics depends on nothing in-house — no Mimir.*, no ManagedAgents.*, no RMA types. Mappers take plain maps and file paths; capture takes Jason-encodable data. Consumers dep on this library, never the reverse. Grep-enforced by test/mimir_analytics/dep_direction_test.exs.

Running the ingest

mix mimir_analytics.ingest \
  --db run_record.duckdb \
  --sessions path/to/session-drops \
  --gateway  path/to/gateway-pull-buffers \
  --evals    path/to/eval_runs

The gateway buffers come from MimirAnalytics.Ingest.GatewayPull (an updated_since-cursor pull of the gateway's read-only Observability API); sessions come from MimirAnalytics.Capture.write/3 call sites in consumers; eval reports are the harness's own JSON artifacts.

Every mapper is idempotent by file contents via ingest_ledger, which keys each ingested file by the sha256 of its bytes, so re-running reads nothing twice. A reused file name with new contents is ingested, except for an eval report: reports are left in place, so a rewritten one is rejected rather than adding its rows again.

The sessions directory is a spool: each file that ingests is moved into <sessions>/.ingested/, which later runs never read. Gateway buffers and eval reports are left in place.

An empty spool file can never be ingested. It is reported once and moved into <sessions>/.quarantined/, next to a <name>.reason file saying why, and later runs never read it there. To re-queue it, move it back into the sessions directory.

An eval report that is empty, or that was rewritten after it was ingested, is rejected but never moved, because other tools read the eval directory. The rejection is recorded in ingest_rejections under the report's content digest, with the reason, so it is reported once per distinct contents; a further rewrite is reported again. To clear one, either restore the file to the contents that were ingested, or accept the new version: delete its ingest_rejections row, and the earlier version's eval_outcomes rows and ingest_ledger row for that file name, then re-run the ingest.

A spool producer must publish each file by rename: write it once under a dot-prefixed temporary name in the same directory, then rename it to its final *.jsonl name. A visible *.jsonl file is then complete and never changes, which is what makes moving it aside after the read safe. Ingest cannot tell a file that is still being appended to from a finished one, so a producer that appends to a published file loses whatever it appends after that file is read.

Upgrading a database

A database created before the ingest ledger was keyed by content holds entries keyed by file name. Those cannot tell a file that grew after it was read from one that was read in full, so mix mimir_analytics.ingest refuses such a database. Rebuild it instead of migrating it:

  1. Stop every writer that appends to a published spool file, and let the last daily file finish.
  2. Delete the database (a local file, or DROP DATABASE on MotherDuck).
  3. Re-run the ingest over the original source directories. Ingest before this change never moved files, so every source file is still in place, and each is read once, in full, into the new database.

Adding a mapper

Fixture first: check a captured real payload into test/support/run_record_fixtures/, write the test asserting exact rows, then the mapper. The fixtures are the wire-shape alarm — when a source changes shape, the diff against a fresh capture is the signal.

Honest gap: managed runtimes

Managed-runtime sessions (provider-hosted loops) have no per-call model_calls rows — their inner-loop calls never traverse the gateway. runs.cost_microdollars is priced from usage upstream at capture; parity comparisons run at session grain. Never synthesize per-call rows.

One-time setup

mix mimir_analytics.setup installs the json and core_functions DuckDB extensions into the local extension directory (network, once per machine/CI runner). Tests themselves never touch the network.

MotherDuck notes

The connection string is the seam: {:motherduck, db} attaches md: using the MOTHERDUCK_TOKEN environment variable and everything else (DDL, mappers, views) is identical SQL. Hard rule: never ATTACH with an empty token — the extension silently falls back to interactive browser auth and hangs the NIF (MimirAnalytics.Store guards this). Tests never touch the network; the single :live-tagged test (excluded by default) exercises the real attach.