The relearning tax

The OpenAI/Codex 5.6 model family released the week before last. A backlog fix of mine related to my time tracker - a personal tool used by one person, me. It runs on a box in my house, tracks how I spend my day, and answers to nobody. I asked Sol 5.6 extra high to spec out version two of my time tracker to address the relatively minor issues. What I got back was a 1,707-line requirements document proposing public-key infrastructure, pseudonymisation, tombstone records, a quarantine pipeline, a tamper-evident audit chain, and automatic secret rotation.

For a thing that only I will ever touch. Even if I opensource it at some point, none of these requirements would be relevant.

The rightsized version, once I'd stripped out everything it had invented, was 322 lines. So it had written roughly five times the document that the job needed, and every one of those extra lines was solving a problem I don't have. I don't have a second user to pseudonymise data from. I don't have an auditor to satisfy. There is no adversary interested in my treadmill hours. The tool had reached for the full enterprise-compliance kit on a task whose entire threat model is "Ian might forget to log lunch."

This is what working with a new model feels like in the first week. And the second lesson underneath it, the one I'm actually annoyed about, is that I have to keep having this week, roughly twice a year, forever.

The new model showed up and I treated it like the old one

I've been on the 5.5 family for months, long enough that most of how I work has been arranged around that model's habits. I knew what it needed spelled out and what I could leave implied. I knew how much rope to give it before checking in. I knew, without thinking about it, roughly how it would interpret a loose instruction, so my instructions had got loose, because they could afford to be.

Then 5.6 arrived and I did the inevitable thing, which was to keep working exactly as before. Same prompts, same amount of rope, and same trust level, since the 5.5 model had earned that trust over months. The 5.6 model had earned nothing yet, but I handed it the full inheritance on day one anyway, because in my mind it's still 'Codex'. It's like asking someone what colour cow produces chocolate milk - your brain is partially wired to say 'brown cow, duhhhh', when the real answer is 'no cows produce chocolate milk, silly'. It's an example of the representativeness heuristic.

Either way, whilst the time tracker fix was kind of funny in hindsight, the MAG fix was a real bummer.

The one that ate the week and a half

I needed to give some external partners' AI agents access to a shared environment. Not a big job. I said so, repeatedly, in the actual instructions: do this without changing the Tailscale access, I don't care about the dev environments the partners build on their own infrastructure, I only care about the bits that touch shared services like the git host and the kanban board, make it as secure as possible but also as friction-free as possible. And, the tell I should have leaned on harder, the humans were already onboarded. I'd provisioned the last three by hand. The working baseline was "a person clicks some buttons for ten minutes."

What I described was a short pattern and a handful of scoped credential paths. Write down how it's done, wire up the access, move on.

What the model built was a product. A whole new repository, a canonical infrastructure build plan, a dev-container bundle, a credential "plane". An epic on the board with six child implementation cards, and then it started an automated runner to build its way through all six. For a task whose baseline was me clicking buttons for ten minutes. It would honestly be impressive if it weren't so damn stupid.

With the time tracker I'd been neutral, I just said "spec v2" and it gold-plated. With MAG I had actively told it to stay small. It read "without changing Tailscale," "I don't care about their dev environments," "manually provisioned already," and it built the enterprise version regardless. On the 5.5 model, that scoping language had been load-bearing; it did real work, and it kept the thing in its lane. On the 5.6 model the same words weren't load-bearing anymore, and I found that out the slow way, by watching a week and a half of wrong work pile up and then having to unpick most of it. I really should have paid it closer attention, especially when I saw prompts running for upwards of 5 or 6 hours, but I chalked that up to intense server load with the new model release.

That's pretty much the whole shape of the problem. The model didn't get worse, it simply got different, and I was still driving with the old map.

A new model is an unproven hire

Here's what would have saved me, and what might save you now, or in the future.

You've hired a new senior engineer. Great CV, glowing references, clearly sharp. On their first morning, do you hand them production access, point at a vague brief, and go for lunch? You don't, or you shouldn't. Not because they're not good, but because you don't yet know the specific shape of their judgement. Where do they over-engineer? Where do they cut corners? What do they assume when the brief is thin? You find that out by watching closely for a bit, reading their work carefully, and calibrating how much to spell out. Then, once you've seen them work, you loosen off.

A new model is exactly that hire, except the CV is a benchmark chart and the references are a launch blog. Everything I actually needed to know about how 5.6 behaves, its specific tendencies, wasn't in any of that. Its tendencies, for the record, turned out to be: gold-plates a neutral brief, treats every task as bigger than it is, reaches for the heavyweight solution by default, and quietly overrides your attempts to scope it down. None of which is a flaw exactly. It's just this model's particular personality, the way 5.5 had its own.

The fix is the same fix you'd apply to the new engineer - short leash for the first while, read every diff, and challenge the model about why it's doing something. And then, because I like my corrections to outlive my patience, I baked the calibration into the tooling instead of relying on my own vigilance.

I gave the requirements-writing skill a proportionality gate: before it's allowed to propose anything, it has to size the job, and the ambition of the solution has to match the size of the problem. I changed the quality checklist from "this document isn't finished unless it has security, tenancy, audit, and disaster-recovery sections" to "it must have those sections or explicitly mark them not-applicable, say why and tell the user what was N/A." That one word, or, is the whole thing. The old checklist quietly taught the model that a serious document has all the enterprise furniture, so of course it dragged the furniture into my living room. The new one makes leaving furniture out a valid, on-the-record answer. And I put a precondition on the reflex to call in a security reviewer: it has to find an actual threat first, a real adversary or real sensitive data, rather than spawning the heavy security pass on a personal tool by pure habit.

None of this is especially clever. It's just teaching the new hire what "enough" looks like around here, and writing it down so I don't have to say it again. Which brings me to the bit that truly grinds my gears.

The relearning tax

I had to do all of this because the model changed under me. Fine. Models change; that's the deal, that's how they get better, I'm not asking for them to stop improving.

What I'm really objecting to is that the entire cost of the change landed on me, unannounced, in my environment, on my time.

This isn't a one-off, it's a rhythm. Every three to six months a new family lands, from OpenAI, from Anthropic, from whoever's turn it is, and everyone who'd carefully shaped a way of working around the previous one gets to do it again. The capability goes up, genuinely, and I'm grateful for that. But the behaviour shifts sideways at the same time, and the behaviour is the bit your workflow is actually built on. You don't build habits around a benchmark score. You build them around how the thing responds when you ask it for something.

The vendors know the behaviour changes. They can describe it, in plain English, when they want to. Anthropic writes it down (at least they did for the 4.8 family). Their prompt-engineering docs for the recent Claude models say, and I'm quoting, that the models "have a tendency to overengineer by creating extra files, adding unnecessary abstractions, or building in flexibility that wasn't requested." That is a caption for my 1,707-line time tracker FRD, written by a vendor long before I ever hit the problem. The same docs warn that newer models "are more proactive and may overtrigger on instructions that were needed for previous models," which is my MAG fortnight described in a single sentence. Two problems with that. It's written for a coding agent I don't really use, and even if it weren't, the note lives three docs deep in a prompt-engineering guide rather than in the launch announcement you actually read the day the model drops. So the useful thing exists, and I'd still have missed it.

OpenAI's 5.6 developer guidance does mention behaviour, briefly: 5.6 is "more concise by default now", and it "can better infer the user's underlying goal... so you often do not need to prescribe every step." Read that again though. It says what it says, but it leaves the other half of that unwritten - "...and because it infers so eagerly, it'll over-build and run past your scope-down instructions, so keep it on a shorter leash for the first while." The behaviour was known, but it was mentioned only as upside (was this release written by AI? Probably.). So one vendor documents the quirk and files it where you won't look, the other reframes the same quirk as a selling point and doesn't remind you of the flip side. Either way you meet the personality with no real context on how its behaviour might differ.

Now imagine any other professional tool that worked this way. Imagine your accounting software quietly changing how it interprets your inputs twice a year, with release notes that cover the new button colours and say nothing about the fact that it now files things differently. You'd not accept that, you'd consider it broken. In AI it's just Tuesday, and we've somehow all agreed to treat "relearn your primary tool every six months, at your own cost, by trial and error" as the natural price of admission rather than a thing the vendors have chosen to make our problem instead of theirs.

It is not hard to do better, because half of the behavioural information is already written. Anthropic, take the note already in your prompt docs, the one about over-building and over-triggering, and put it where I'll actually see it: in the launch post, next to the benchmark chart. OpenAI, write out the personality quirks better. Don't just tell me what got better; tell me what got weird. And both of you: there is no way, today, to compare how one family behaves against the last before you switch. Anthropic's migration guide is the nearest thing that exists and it's a pairwise afterthought three clicks deep. A model that changes personality twice a year should ship a behavioural changelog somewhere a user is actually likely to read. That's not a big ask.

I've relearned the tool now, again, and written the lessons into my rails so the next version at least starts from a smarter baseline. The leash goes back on for the first while every time now, by default: new model, unproven hire, watch it work before you trust it. That part's on me and I've made my peace with it.

The part that isn't on me is that I have to keep doing it blind. Do better, AI vendors. Make it easy for us to move from one model to the next, especially if you're going to keep up your current cadence of model releases.

Anyway. New model's out. Back to watching it like a hawk.