Agent Self-Improvement

A text-to-SQL agent that detects its own degradation and learns from its failures, no human in the loop.

AI Engineer World's Fair Hackathon 2026 (Cerebral Valley, San Francisco) · June 27, 2026 · 24 hours · The Self-Improvement Stack track · Team of four

Architecture at a glance

Agent Self-Improvement architecture Spider queries flow into the Harness, which emits TelemetryRecords to the Detector. The Detector emits DriftEvents to Correction. Correction writes few-shot examples back into AgentConfig, which the Harness reads. Every stage appends to events.jsonl, which the Viewer turns into a recovery curve. feedback → AgentConfig.few_shot_examples Spider queries Harness agent runs on Spider, execution-accuracy eval Rohan TelemetryRecord Detector windowed drift detection WARMUP→NORMAL→DRIFTING Logan DriftEvent my stage Correction teacher generates + verifies SQL → few-shot Mihir events.jsonl · append-only event log, written by every stage Viewer events.jsonl → recovery curve Yiwen
Agent Self-Improvement architecture: Spider queries, Harness, Detector, Correction, feedback into AgentConfig, events.jsonl, Viewer Spider queries Harness agent runs on Spider, execution-accuracy eval Rohan TelemetryRecord Detector windowed drift detection WARMUP→NORMAL→DRIFTING Logan DriftEvent my stage Correction teacher generates + verifies SQL → few-shot examples Mihir feedback: AgentConfig.few_shot_examples every stage appends events.jsonl append-only event log Viewer events.jsonl → recovery curve Yiwen
  • Frozen Pydantic contracts between stages (TelemetryRecord, DriftEvent, CorrectionAction, AgentConfig). Each stage was built and tested in isolation against mock fixtures, then wired together by one orchestrator.
  • Append-only event log. Every stage writes to events.jsonl; the Viewer only reads it.
  • The loop is data, not weights. Feedback is concretely an edit to AgentConfig.few_shot_examples, a list that starts empty and grows as Correction appends learned examples. Nothing is fine-tuned, nothing is reverted, and no bigger model is swapped in.

The problem

Production AI agents drift. The input distribution shifts, accuracy slides, and nobody notices until a user does. Most “self-improving agent” demos can’t tell a genuinely bad run from noise, and few check whether the improvement survives a real test.

In 24 hours our team of four built a self-correcting text-to-SQL agent that detects its own degradation, learns from its failures, and holds the improvement to a real statistical bar. I owned the Correction stage: the part that turns a detected failure into something the agent actually learns from.

The architecture

Four typed stages with frozen Pydantic contracts between them, over an append-only event log. Harness runs the agent on Spider questions and emits TelemetryRecords. Detector consumes those and emits DriftEvents. Correction (mine) has a stronger teacher model generate and verify SQL to mint few-shot examples. Viewer turns the event log into a recovery curve. The feed starts on easy questions, then shifts at a change-point to hard and extra-hard ones: the detector catches the drop, Correction turns the agent’s own failures into examples, and the agent recovers on hard questions it has never seen.

What was hard, and what I decided

Corrections you can trust. A teacher model’s output is a guess, not a label. For each failing case, the teacher (MiniMax-M3) generates corrected SQL from the question and the database schema. I accept it only if it executes to the correct result set against the gold query (the Spider execution-accuracy metric). If it doesn’t, the teacher SQL is discarded and the gold SQL is used instead. Teacher quality affects yield, not correctness: every example that reaches the agent is known to be right.

Not overfitting to the drift window. The easy way to “improve” is to teach the agent the exact questions it will be graded on. Examples are stratified out-of-sample by database: hard questions are split into a disjoint LEARN pool, the only source of examples, and a HELD-OUT pool that serves as the benchmark. The agent can’t regurgitate an example it was handed.

The second risk is forgetting. Filling the prompt with hard multi-join queries can break easy queries the agent already got right, so every correction also carries a few anti-forgetting anchors (easy successes). It held: of the 7 held-out questions whose outcome changed, 7 improved and 0 regressed.

A test that actually means something. If the base model already knows the answers, “learning” is just prompt luck. We deliberately ran a weak base agent (MiniMax-M2.7-highspeed) against a stronger teacher, so the base genuinely struggles on hard SQL and any gain has to come from what it learned from its own failures.

The rest of the loop. The other stages are my teammates’ work. Logan’s detector is a windowed WARMUP → NORMAL → DRIFTING state machine that fires only when the windowed mean stays below baseline for several consecutive records, and filters transient API outages so they can’t fake drift. Rohan’s harness streams the change-point feed with execution-based telemetry. Yiwen’s FastAPI + Chart.js viewer renders the recovery curve live from the event log.

Results

In one unattended run the detector fired at the change-point (severity 0.295, windowed accuracy 0.48 vs a 0.775 baseline), Correction synthesized 10 few-shot examples (3 teacher-verified, 5 gold-fallback, 2 anti-forgetting anchors), and the agent recovered.

+0.267
hard-bucket accuracy gain
p = 0.016
paired McNemar exact test
7 / 0
questions improved / regressed
0.295
drift severity at the change-point

Hard-bucket execution accuracy, same 30 held-out questions, same eval:

The self-improved agent beat its own teacher. Half of its examples are execution-verified gold SQL that the teacher couldn’t produce, which scaffolds the weaker model past the stronger one’s unaided accuracy.

Is it chance? We re-evaluated the same 30 hard questions with and without the learned examples and ran a paired McNemar exact test: p = 0.016 (two-sided), paired delta +0.233, 95% CI [+0.08, +0.39]. The model isn’t perfectly deterministic even at temperature 0, so absolute accuracy moves by 2 or 3 questions between runs (the demo run measured +0.267, the paired test +0.233). The claim I’d stand behind is the paired delta with its interval.

Tech stack

Python · Pydantic (typed contracts) · SQLite + Spider execution-accuracy eval · MiniMax (OpenAI-compatible API) for the base and teacher models · FastAPI + Chart.js (viewer)

Team

A four-person team, equal contributors, one stage each:

  • Rohan: Harness / telemetry
  • Logan Rudd: Detector / drift detection
  • Mihir Agarwal (me): Correction / learning
  • Yiwen: Viewer

Logan wrote up the project from the detector’s side: loganrudd.com/agent-self-improvement. The code lives at mihir-agarwal0211/agent-self-improvement, copied with full history from the original repo, which Logan hosts.