The guardrails were wrong at version one

Another week, another example of a security check I thought was working but wasn't. The check had fallen over before it checked anything. I only found out when I went in to test it properly, fed it exactly the thing it was supposed to stop, and watched it wave the thing straight through. It was this, in action:

By now I've written more than once about how a good AI agent will do the wrong thing at speed and swear blind it went fine, so I'll spare you the reprise. What I haven't written about is what happens when the thing doing the wrong thing at speed is the guardrail you built to catch the agent. That's a whole new flavour of fun.

People online talk about AI guardrails as though writing one is the win, when it really isn't. Every guardrail I depend on today is version three or four of something that was wrong at version one, and the fact that they were broken in V1 was rarely obvious. It bears repeating: a control that's subtly broken is more dangerous than no control at all, because now you're relying on it. That changes the real win from 'I wrote the guardrail, job done, yay me!' to 'I wrote the guardrail and then I reviewed it. A lot. Yay me!'

For your delectation, two examples from my recent memory where I lost track of this win condition and didn't review my guardrails well (or at all) and inflicted suffering on myself. And then the pattern I've now put in place to make sure I suffer less in the future.

The guard that had never worked

I write hooks that sit between the agent and the things it can do. One of them scans for secrets: before a command runs, it checks the payload for the shapes of things that should never hit disk or land in a transcript. API tokens, cloud keys, private keys. The private-key check looked for the header every PEM-encoded key starts with, that -----BEGIN ... PRIVATE KEY----- line. Simple, obvious, done, yay security!

Version one didn't work even once.

Here is the actual bug, because it teaches something you can use. The check ran a pattern that began with a dash:

grep -q '-----BEGIN.*PRIVATE KEY'

grep reads a string that starts with - as a set of options, not as a pattern to search for. So instead of looking for private keys, grep tried to parse -----BEGIN... as flags, failed, errored out, and exited non-zero; which the surrounding code read as "no match found." Every single time. The fix is two characters, an -e to say the next thing is a pattern, not a flag:

grep -qe '-----BEGIN.*PRIVATE KEY'

Two characters. But sit with what those two characters were doing while they were missing. I'd written a security control, wired it in, watched it not complain, and decided I was protected. It was reporting "all clear" not because it had checked and found nothing, but because it had fallen over before it checked anything at all. It really is the worst failure a guardrail can have, and violates all sorts of good coding practices, let alone security and governance practices.

This is the same green-light problem I keep coming back to, the one behind silent failures and every intermittent test that passes for the wrong reason. "It didn't complain" is not "it worked." With a security check, the gap between those two is the entire risk, and I'd been gently marinading in it without knowing.

The lesson from this is two parts. 1) Write better regexes. 2) In God we trust; the rest we verify. So now I make my controls prove they can catch the thing they're meant to catch and I feed them the bad input on purpose and check that they bite. After all, who wants a guard crocodile that naps all day, every day?

The guard that cried wolf

The opposite failure is just as painful, and I built that one too.

I had a monitor watching my own configuration for tampering, the kind of confused-deputy protection that matters when an agent with broad access is loose in your environment. It was meant to fire when something security-relevant changed underneath me. Trouble is, it fired on everything. Ordinary, harmless changes I made a dozen times a day set it off, so within a week the alert meant precisely nothing. I'd see it, figure it was almost certainly the benign thing again, and wave it past. Which is exactly how you miss the one time it bloody well isn't. After all, an alert you've trained yourself to ignore is worse than no alert - and this is called alert overload. It costs you attention and gives you nothing back, and it leaves you feeling watched while you aren't.

The fix wasn't to make the check louder or stricter. It was to narrow what it watched down to the things that actually matter, so that when it fires now, it means something and I actually look to see what happened. The signal came back the moment I stopped drowning it.

Just so it's clear I'm not being obtuse, I'm deliberately not going to hand you the exact set of things it watches. That's the part that would help someone map my defences, it earns you nothing, and it may not apply to your development environment anyway. The transferable lesson is the one that matters: make your control alert you, just don't make it over-alert you. This means making sure you're feeding your AI agent with a good prompt to build your guardrail, then looking at the output and making sure it's not filling your screens with meaningless errors.

Teaching the agent to check its own work

Given my enduring suffering (there are many more examples than the two I pointed out), I built something to make sure I have fewer V1 issues with my guardrails. Happily, it also turns out that this means fewer V1 errors with other agent outputs too. It does consume more tokens, but I take this as an upfront cost that reduces much higher future cost.

Like I said, left alone, an agent hands back its first attempt as though it were finished and sometimes it is. Often it's only most of the way there, with a gap it would have caught if it had looked again with fresh eyes. So I have it look again: complete the task, then run its own review-and-fix pass against the actual output, checking for the things that actually go wrong, missing edge cases, broken wiring, a claim it didn't verify, a regression outside the intended change. Find the material problems, fix them, validate the fix, and only then hand it back.

Just so we're clear, the naive version of this is worse than useless and two failure modes will show up immediately. First, it will never stop. An AI agent review loop with no brake will always find something to change, so it churns, polishing wording and reshuffling things that were fine, mistaking motion for improvement. Second, it will scope-creep: handed a small task, it will "notice" adjacent things and start rewriting them, which is how a two-line fix turns into a diff you can't safely review.

Both fixes for these are judgment encoded as a rule. The loop stops at diminishing returns, when a pass turns up nothing material or only cosmetic preference. Tedium or complexity is not a reason to stop, but "the last pass found nothing that matters" is. And then to make sure there's a sane stop, I also hard-cap it at three loops by default. I can modify that in the instruction I give (say, five loops), but having a hard stop is important. And I also hold it to material issues only: fix what's actually wrong or what creates a real regression risk, and leave the rest alone, because a review pass is not a licence to redecorate. It's a coding agent, not a code-related-interior-decorator. The benefit of this loop approach is that it enforces the line between a problem and a preference, which is the same line a good senior reviewer holds and a junior one (or AI agent) doesn't.

The rails are never finished

The thread running through all three of these is one I only really understood once the review loop existed: the guardrail deserves exactly the same suspicion as the agent it's guarding. I built the loop to catch the agent's V1 mistakes, and then watched it start catching my V1 mistakes too, the broken secrets check and the trigger-happy monitor included. It turns out I'm not a more reliable author of guardrails than the agent is of code, I just used to find out later.

So now I take my own medicine and treat my own controls the way the loop treats the agent's output: assume it's wrong until I've watched it be right. To summarise:

1. A guard has to be shown catching the thing it guards against.
2. An alert has to be precise enough that it doesn't drown me in an ocean of noise.
3. A review loop has to know when and why to stop. And it needs to exist in the first place.

The promise people keep selling is autopilot: hand it over, walk away, let the robot cook. My many trials, most of them errors, say that's not the model that works. What works is the centaur (thanks, Cory Doctorow): the machine does the work it's extraordinary at, and the person holds the judgment about what "done" and "safe" and "enough" actually mean. You encode as much of that judgment as you can into the rails the agent runs inside. And then you stay humble enough to remember you probably got the rails wrong the first time, because you almost always did.