Every time you correct your AI coding agent, you're generating training data, but are you just throwing it away?
"You need to run CLI commands non-interactively." "Try again but split it into smaller calls." "Stop — I already told you to use the staging bucket." These corrections are signals. They reveal gaps between what the agent knows and how you actually work. The problem is that most agent setups treat each session as disposable. The agent makes the same mistake on Tuesday that you corrected on Monday.
What if the agent could learn from those corrections — not through fine-tuning or retraining, but by updating its own configuration?
The configuration hierarchy
Kiro CLI organizes agent behavior through a layered configuration system:
Steering docs — universal behavioral rules that apply to all agents (e.g., "always use non-interactive commands," "pin dependency versions")
Skills — domain-specific knowledge packages (e.g., AWS CLI best practices, Docker build patterns)
Agent prompts — per-agent instructions that define personality, capabilities, and constraints
Hooks — scripts that fire at agent lifecycle points to enforce policy, collect data, or block unsafe operations (e.g., reject unpinned dependencies, prevent unauthorized config changes)
When an agent does something wrong, the root cause usually maps to one of these layers. It either violated a rule that doesn't exist yet, lacked domain knowledge for the task, or has agent-specific instructions that are too vague.
The flywheel concept
The flywheel is a stored prompt that turns session history into configuration improvements. It works in five phases:
Phase 1 — Session analysis. A stop hook logs a lightweight summary after every agent turn to ~/.kiro/flywheel-log.jsonl — timestamp, working directory, and a response preview. The flywheel reads this log first for a fast index of recent activity, then dives into the full session JSONL for correction details: explicit corrections, redirections, cancelled turns followed by rephrased requests, repeated instructions, and frustration signals where the user simplifies their request after a failed attempt.
Phase 2 — Pattern recognition. One-off mistakes get filtered out. The agent groups corrections by theme — "output too verbose," "wrong tool choice," "hallucinated API" — and focuses on patterns that appear across multiple sessions.
Phase 3 — Cross-reference. For each pattern, the agent reads the existing steering docs, skills, and agent prompts to determine whether the issue is already covered (but too weakly worded), completely missing, or being ignored.
Phase 4 — Propose changes. The agent writes a structured report with evidence (quoted user messages), root cause analysis, and draft content for each proposed configuration change. Each proposal is classified as a steering update, new skill, or agent prompt modification.
Phase 5 — Interactive review. The agent walks through each proposal. You approve, modify, or skip. Approved changes get applied to the correct files with proper formatting. Nothing ships without your sign-off.
What correction events look like
Kiro stores session data as JSON metadata paired with JSONL conversation logs. Each line in the log is a typed event — user prompts, assistant messages, and tool results. The flywheel scans user prompts for patterns that indicate the preceding assistant action was wrong:
| Signal |
Example |
| Explicit correction |
"No, I meant the production account" |
| Redirection |
"Instead, use the Converse API" |
| Retry after cancel |
User cancels a turn, then rephrases the same request |
| Repeated instruction |
User restates something from two messages ago |
| Simplification |
User breaks a complex request into smaller pieces after a failure |
The key insight is that the correction message and the assistant message before it form a pair: what the agent did wrong, and what the user wanted instead. That pair, abstracted into a general principle, becomes a candidate configuration rule.
A concrete example
Across several sessions, the agent generates tool calls with payloads that exceed size limits, forcing the user to say "try again but split up the work." This pattern appears three times in a week.
The flywheel identifies the pattern, checks existing steering docs, finds no rule covering output size management, and proposes a new steering doc:
---
inclusion: always
---
# Output Size Awareness
When writing content through tools (MCP servers, file writes, API calls), check whether the target has size constraints. If the payload is large,split it into multiple smaller operations proactively — don't wait for a failure or user correction.
You review it, tweak the wording, approve. The agent applies it. Next week, the agent chunks large writes automatically.
Why this works
The flywheel works because it operates on the same configuration layer the agent already reads. There's no model retraining, no fine-tuning pipeline, no vector database to maintain. It's just markdown files that get included in the agent's context on every session.
It also works because it's conservative by design. The prompt requires patterns across multiple sessions before proposing a change, quotes actual user messages as evidence, and never applies changes without explicit approval. A preToolUse hook called the "drift guard" enforces this — it blocks any write to steering, skills, or agent config files unless the user has approved the change. You're not handing the agent a blank check to rewrite its own instructions — you're reviewing pull requests against your agent's behavior, with a guardrail that prevents unauthorized self-modification.
The compound effect is what makes it a flywheel. Each correction you make today becomes a rule that prevents the same class of error tomorrow. Over weeks, the agent accumulates your preferences, your team's conventions, and your project's constraints — not as ephemeral context that disappears when the session ends, but as persistent configuration that shapes every future interaction.
Getting started
The implementation is a stored prompt file plus three hook scripts. The stop hook collects turn data passively. The drift guard hook prevents unauthorized config changes. The dependency pinning hook is a bonus — it blocks writes to package.json or requirements.txt with unpinned versions, demonstrating how hooks enforce policy beyond the flywheel use case.
For example code, check out this AWS Sample I published with my Kiro CLI configuration. If you just want an example of the call hook code, I also saved it as a GIST.
Run the flywheel periodically — weekly works well — or whenever you notice the agent repeating a mistake you've already corrected. Each run produces a report, and each approved change makes the next run's report shorter.
The best agent configuration isn't the one you write on day one. It's the one that evolves from how you actually use the tool.
