agent-runtime
Agent Runtime
Hand an AI assistant real tools, like email or a calendar, and you need a record of what it did. This builds that record: every step is written to a log, so a whole run can be replayed exactly; risky actions pause for human approval; and a crash resumes without repeating work. 93 tests hold it together, all run against a scripted stand-in model, never a live AI service.
The Anthropic and OpenAI adapters are implemented against their current APIs and covered by offline contract tests, but no live API calls were made because no keys were available. Everything else, including the full test suite, the CLI, and the web UI, runs and was verified offline.
Overview
An event-sourced runtime for tool-using LLM agents, with the operations layer that agent demos usually skip: deterministic replay, crash-safe resume, an approval gate for side effects, config-driven failure injection, and a YAML scenario suite that regression-tests agent behavior between model versions.
The agent loop itself (plan, call a tool, observe, continue) is a weekend's work. What makes agents deployable is everything around it: what exactly happened in a given run, can we reproduce it, did the agent email a customer before anyone approved it, and does the new model version still pass the flows that worked last week.
This project treats those as the core problem. Every run is an append-only event log, and every feature (replay, resume, approvals, cost accounting, regression diffs) is a function of that log.
Key features
- An agent loop over a ModelAdapter interface with a deterministic mock adapter plus Anthropic and OpenAI adapters.
- A typed tool registry with JSON-schema parameters, a side-effect flag, timeouts, and retries with backoff; shipped tools include sandboxed filesystem access, allowlisted HTTP GET, a CSV updater, an email drafter that writes .eml files and never sends, and a SQLite calendar.
- Event sourcing: an append-only SQLite log per run, with one projection as the single interpreter of what a log means.
- Deterministic replay that re-drives the loop purely from recorded turns with no network, compared byte for byte, with --until for stepwise state.
- Checkpoint and resume without duplicate side effects, using idempotency keys derived from the log.
- An approval queue that pauses side-effecting calls, driven from the CLI or a FastAPI UI with diff-style previews, plus failure injection proving the loop degrades cleanly.
Results
- 93 tests, all offline, covering replay, resume, approvals, faults, and the scenario suite.
- 24 YAML scenarios across 3 example agents; the regression runner prints pass/fail with a first-divergence diff per scenario and an HTML report.
- Replay is byte-identical across happy paths, denials, faults, and failed runs.
- A resume test crashes a run at the worst instant and asserts a side-effect counter stays at 1.
Tech stack
Challenges
- Byte-identical replay fought every source of nondeterminism: the store takes an injectable clock, sleeps are injectable no-ops, and model_requested records a digest rather than anything time-dependent.
- The mock adapter's script index and call counter had to be different numbers; splitting them fixed fault plans and turned out to be exactly what resume needed.
- Closing the duplicate-side-effect window took ordering, not locks: the executor writes the result to an INSERT OR IGNORE ledger before the runtime appends the event.
What we learned
- Deriving all state from a projection collapses feature count: resume after crash, continue after approval, and a plain next iteration became the same function.
- Determinism is a budget you spend field by field; any payload value not a pure function of the log is a future replay divergence.
- Separating expected tool failures from unexpected ones buys a retry policy for free.
- A per-event type-and-detail signature is enough for useful regression diffs with no diffing library.