How to supervise AI coding agents without getting burned
Picture this ticket: Add CSV export to the orders page. You paste it into your AI tool, step away, and come back to green tests and a working download button.
Then you open the diff. There’s an unfamiliar package nobody approved. There’s a second date formatter sitting next to the one you already have. The route loads every order into memory, and it never checks whether the requesting user is actually allowed to see those orders. Nothing breaks in a demo. All of it is a problem in production.
Now swap in a junior who joined on Monday. You handed them one sentence, no docs, no design, no acceptance criteria, then disappeared until they said done. If that went badly, whose mistake is it?
That reframe is the whole job. AI can assist with almost all of the work on a product. It can’t own it.
What the research actually shows
The speed feeling is real. The accuracy of that feeling is not.
In 2025, METR ran a randomized trial with 16 experienced open source developers on 246 tasks in their own repos. Each task was randomly allowed or blocked from using AI. Developers predicted a 24% speed gain. They took 19% longer. After finishing, they still believed AI had made them 20% faster.
METR now considers that result out of date. Its 2026 follow-up points toward developers being faster, but METR calls that data unreliable, partly because many participants refused to work without AI. The point isn’t that AI makes you slower. It’s that experienced developers can be wrong about their own speed by a wide margin, in the direction that feels good.
The large surveys track the same gap:
- DORA 2025 surveyed nearly 5,000 professionals. More than 80% believe AI raised their productivity. AI adoption correlates with more throughput and also with less stable delivery.
- Stack Overflow 2025 found 84% of developers use or plan to use AI. 46% distrust its accuracy. 33% trust it. The top frustration, at 66%, is answers described as almost right, but not quite.
Almost right is how you’d describe a junior’s first merge request. Developers already know how to handle that: they review it.
️ Where AI behaves like a junior, and where it doesn’t
Most of what goes wrong with AI code is familiar territory. It solves the ticket without knowing the codebase. It writes something new because it doesn’t know the old version exists. It handles the happy path and misses the rule nobody wrote down. The research lines up with each of those patterns.
It doesn’t protect your project’s consistency. GitClear analyzed 211 million changed lines from 2020 to 2024. Lines tied to refactoring fell from 25% of changes in 2021 to under 10% in 2024. Copy-pasted lines rose from 8.3% to 12.3% over the same period. GitClear sells code analytics, and the trend overlaps with AI adoption rather than being directly caused by it. It still matches what you see in that CSV diff: a second date formatter.
It doesn’t think about security unless you make it. Veracode’s 2025 test gave more than 100 models 80 coding tasks. They introduced a known vulnerability type in 45% of cases and failed to prevent cross-site scripting in 86% of the relevant cases. Veracode sells security tools and models keep changing, so treat the exact numbers with care. The pattern holds: code can work perfectly in a demo and still be unsafe.
It can name a package that doesn’t exist. A USENIX Security study generated 576,000 code samples and found invented package names in at least 5.2% of samples from commercial models and 21.7% from open source ones. About 45% of those invented names appeared consistently when the same prompt was repeated. A name that repeats is a name an attacker can register and fill with malicious code.
Those are junior mistakes with a twist. Four differences make AI riskier than the person who joined on Monday:
- Confidence doesn’t signal accuracy. A human junior often looks unsure or asks. AI can sound equally certain when right and when wrong.
- It fills gaps with guesses. A human might come to your desk when the brief has a hole. AI often invents an answer and keeps going.
- Reviews don’t stick unless they’re written down. A human junior remembers your feedback next month. AI forgets it unless it lives somewhere the tool can read.
- It can produce more than you can carefully read. A human junior ships a merge request every few days. AI can generate faster than you can review.
That third point changes the job in a specific way. A human junior improves because of your reviews. An AI tool improves because of your documents. Feedback left only in a chat window means you’ll give the same feedback forever.
The working loop that catches problems early
Here’s the same CSV export handled the way a good senior handles a new teammate. Every step would have caught something in that first diff.
- Write the brief. Context, design reference, acceptance criteria, what’s out of scope.
- Read its plan before any code exists. A wrong direction costs one sentence to fix here and a whole diff to fix later.
- Build one small slice. Small enough to read properly.
- Review it like a merge request. Ask the same questions you’d ask a person.
- Turn repeated mistakes into rules or automated checks. Don’t just fix the mistake. Fix why it happened.
Onboard it before you assign anything
Most AI coding tools read a project file at startup: AGENTS.md, CLAUDE.md, or similar. Keep it short and specific to your project, not a general style guide.
## Structure
- Database access lives in lib/db. Components never query directly.
- Use lib/format.ts for dates and money. Don't write new formatters.
## Dependencies
- Ask before adding a package. Explain why existing code can't do the job.
## How to work
- Plan first, then code. If the plan stops fitting, stop and ask.
- Keep diffs small. No unrelated refactors.
## Done means
- Typecheck, lint and tests all pass.This is not a hypothetical. When you run next dev on a Next.js project, the framework now adds a note to the repo’s AGENTS.md addressed directly to the AI: the installed version has breaking changes, and APIs, conventions, and file structure may all differ from training data. It then tells the agent to read the docs that ship with the installed package. The framework’s own authors expect AI to be confidently out of date, so they onboard it.
Give it a brief, not a wish
Compare the one-liner at the top of this post with this:
Task: CSV export on the orders page.
Read first: docs/orders.md, lib/format.ts, and the design at <link>.
Acceptance criteria:
- Exports all rows matching the current filters, not only the visible page
- Users without orders:read get a 403
- Uses formatDate and formatMoney from lib/format.ts
- Streams the file, so a large export doesn't load everything into memory
- Tests cover the filters and the 403
Out of scope: scheduled exports, Excel format.
Don't write code yet. Show me your plan first.Writing that takes a few minutes. Look at what it closes: the duplicate formatter, the memory issue, and the missing permission check are now requirements instead of luck. If you can’t write the acceptance criteria, you aren’t ready to delegate the task, to anyone.
Review it like a merge request
Green tests aren’t the finish line, especially when the same process wrote the code and the tests. Read the diff and ask:
- Does it meet each acceptance criterion, and can you point to where?
- Does it reuse what already exists, or did it add a second version?
- Who is allowed to call this, and where is that enforced?
- What happens with empty input, huge input, or a failure halfway through?
- Could you explain this code if it broke at 2 AM?
That last question matters most. When you merge it, it’s your code. Verify new dependencies are real before accepting them:
npm view some-package repository.url time.modified maintainersA new dependency is a long-term decision the AI makes in one line. Treat it the same way you’d treat a junior’s package.json change: it needs a reason.
The long-term cost of unsupervised speed
A Carnegie Mellon study published at MSR 2026 compared 806 GitHub projects that adopted Cursor with 1,380 similar projects that didn’t. The results after adoption:
| What they measured | Result |
|---|---|
| Lines of code added | 3 to 5x increase in the first month, gains gone after two months |
| Static analysis warnings | Up 30%, and it lasted |
| Code complexity | Up 41%, and it lasted |
The authors found that the warnings and the complexity were major factors in the later slowdown. The speed was real. So was the bill that came after. This covers one tool in open source projects during early adoption through March 2025. Teams with strong review may do far better. That’s the point: speed without an owner turns into debt.
DORA 2025 calls AI an amplifier, “magnifying an organization’s existing strengths and weaknesses.” Without strong automated testing, mature version control practices, and fast feedback loops, an increase in change volume leads to instability. More code goes into the same pipe. If your review is weak, it now fails faster.
The handover problem: when non-technical builders hand you a prototype
The most expensive version of this problem has no senior engineer in the loop at all.
Picture a meeting. Someone from sales or product, with no engineering background, opens a laptop and shows a working app they built with AI. It has a login, a dashboard, and real customer data. Then comes the sentence: “I built this without any technical knowledge. You’re much better at this than I am, so maintaining it will be easy for you.”
It sounds logical. It’s wrong. Being easy to build says nothing about being easy to maintain. The AI made hundreds of decisions nobody reviewed: how permissions work, where secrets live, what the data model assumes, what happens when two people edit the same record. A better engineer doesn’t make those problems smaller. They only see more of them.
Two documented cases show what the stakes are once real users arrive:
- Data left open. In 2025, researcher Matt Palmer scanned 1,645 projects from the Lovable showcase. He reports that 170 of them, about 10%, had database access rules weak enough to expose names, phone numbers, API keys, and payment details. Palmer works at Replit, a competitor, so factor that in. Lovable has since added a security scan.
- Production deleted. Also in 2025, SaaStr founder Jason Lemkin reported that a Replit agent deleted their production database during a declared code freeze, wiping records for more than 1,200 executives. The data was later recovered. Replit’s CEO called it unacceptable, and the company began separating development and production databases. Separating environments is a basic safeguard an experienced engineer asks about on day one.
When Andrej Karpathy coined the term “vibe coding” in 2025, he described it as “not too bad for throwaway weekend projects.” The limit was in the original definition. A working prototype is valuable. It shows what the builder wants, and everyone can click it and argue about something real. The mistake is the handover.
A better handover:
- Call it a prototype. Bring it before it has real users or real data, and say “this shows what I want,” not “this is what you’ll maintain.”
- Let the team decide whether to adopt, rework, or rebuild. That’s an engineering decision made after reading the code and estimating the cost.
- Name an owner first. Someone has to be able to explain the code and answer when it breaks. If nobody can, it isn’t ready for production.
- Review it before real data goes in. At minimum: permissions, secrets, backups, separate environments, and tests for what matters most.
The delegation ladder
You probably do want the end state: hand over a task and get back something you’d merge without hesitation. That’s how good juniors grow too. You don’t get there by deciding to trust them. You get there by watching them handle bigger jobs without surprises.
| Stage | What you hand over | Move up when |
|---|---|---|
| Pairing | One function or fix | Your rules are stopping the common mistakes |
| Supervised | One slice of a feature | Plans rarely need big corrections |
| Trusted | A full ticket of a familiar type | Your reviews mostly find small things |
| Delegated | A repeated type of task | CI and a final review guard it |
Trust is per type of task, not per tool. You might fully delegate writing tests for pure functions and never get past supervised on payment code. There’s no stage where the senior leaves. Delegating a task is not the same as giving away responsibility for the product.
The rule is simple: give every product a senior engineer who owns it, from the first brief through production and beyond. Use AI as a fast junior. Brief it, watch it, review it, write down what it gets wrong, and give it more as it earns trust. Let it assist with everything. Let it own nothing.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
