Agent Failure Detection: Checks Beat Monitors
A deterministic recompute-and-coverage check caught agent failures at zero false positives where a learned monitor hit 17%. What to ship first.
The blog is where we publish benchmark methodology, architecture decisions, and product thinking for people who actually need to evaluate the system.
Long context windows are useful until they become expensive, inconsistent, and impossible to reason about. Checkpoints give agent systems a cleaner recovery model.
A deterministic recompute-and-coverage check caught agent failures at zero false positives where a learned monitor hit 17%. What to ship first.
Tool poisoning becomes durable when untrusted tool output is consolidated into an agent's long-term memory. Provenance-aware writes close that gap.
Retrying failed calls is safe for reads. Without an idempotency contract, the same default turns a transient write failure into permanent bad state.
A new paper shows agent context costs grow quadratically, not linearly — and why chaining more agents doesn't fix reliability either.
Timestamps only tell you a record is old, not whether it is wrong. Most freshness bugs in agent memory come from write races, not stale clocks.
A single vector index with one decay curve cannot tell a durable preference from a one-off event. Typed records fix that at write time.
Four hardening PRs reveal the defaults metronix-memory holds contributors to — miss them and your first PR bounces on review.
AWS and Microsoft committed $3.5B to embedded AI engineers. What the deployment gap means for enterprise agents, and why the fix is a context layer.
A Berkeley team gamed 8 top AI agent benchmarks without solving a task. What broken evals mean for how enterprises pick and ship agents.
The "N-hour time horizon" is a difficulty score, not a runtime. Here's why the 80% reliability gap is where production agents fail — and how to close it.
A benchmark is only useful if it survives contact with real workloads. The trick is to optimize for reproducibility before bragging rights.
Developers can smell fake openness from orbit. If the repo is real, the docs, issues, and architecture boundaries should be real too.