Skip to content

Interim Review and System View: Taking Stock of the Series

Months after the conversation where it all began, it’s time for an honest assessment: which expectations from the opening came true, which didn’t, what remains open – and why the most exciting question is no longer what the models can do, but what organizations make of it.

≈ 10 min readSoftwareAIMethodology

The conversation with which this series began took place in October of last year; this text takes stock as of August 2026. The opening ended with a resolution: to distrust my own enthusiasm for the time being, look closely and measure every result against more than thirty years of craft. The last installment announced that it would take this resolution at its word. So: which of the expectations came true, which didn’t, what remains open – and what system view has emerged from the individual sessions of the beginning?

Part 13 of the series “From Session to System” – the opening post is here.

The enthusiasm survived the test – barely#

The most important thing first, because everything else builds on it: the few days of trial in October have become everyday work, and that is the least spectacular evidence I can offer. I no longer think about these tools, any more than I think about the editor or version control. They are infrastructure. A flash in the pan looks different – you recognize one by the fact that after three months you’re telling stories about it again instead of working with it.

Still, it would be too cheap to simply declare my own perception to be the assessment. In July 2025, the American research institute METR – a nonprofit that measures the capabilities of AI systems with controlled experiments – published a study that has occupied me more than any benchmark: experienced open-source developers were measurably slower with AI support on randomized tasks – by roughly a fifth – and yet afterwards estimated that they had been considerably faster. So exactly the perception I have relied on since the opening can be systematically deceptive. I take that seriously, because it is the same warning I gave myself in the opening: fresh enthusiasm is a poor witness.

That’s why I check against things that can’t be talked up. What is actually done in the evening? Which of the stalled projects are in operation today? How often did I find errors in reviews that would have slipped through without the shift toward checking? By this standard, the assessment is positive – as an observation, not a measurement; I lack reliable numbers on it. That is exactly why the study remains important for my assessment: as long as I don’t measure with comparable rigor, I have to factor in that my perception may deceive me too. The study does, however, measure tasks that were familiar to the developers, in code they knew well. My biggest gain lies elsewhere – in the projects that would never have been started at all without these tools. That doesn’t refute the study; it’s a different calculation. Both can be true at the same time.

What came true#

The opening recorded three observations. All three have survived the months since the opening, but none has stayed the way it was.

The first – generating gets cheap, checking becomes the bottleneck – has turned out to be the load-bearing observation of the entire series. In hindsight, almost every installment is an answer to this one shift: the harness organizes checking around the model, the house rules defend it against my own convenience, grounding anchors it in real sources, and rules in the path make it independent of the checker’s form on the day. What I didn’t see yet in November: the bottleneck keeps moving. First checking was scarce, then cleanly writing down what is supposed to apply – by now it is deciding what gets built at all.

The second observation – the drawers open up again – came true, with a limitation I didn’t see coming back then. Some of what had been lying around for years is in operation today: smaller than the original idea, and that is not a defeat but a finding. Because what the tools don’t deliver is a judgment on which of the old ideas were worth it. I shut down two projects again after a few weeks – failed faster than would have been possible before, and that is exactly the underestimated gain: cheap implementation makes discarding cheap too.

The third observation – the tone is always right, even when the content isn’t – was a gut feeling in November. It has since become a structural explanation and then a procedure. The distrust has stayed, but it has moved: from my private vigilance into check steps that don’t depend on whether I’m having a good day. That is perhaps the most compact summary of what these months have done to my way of working – diligence went from being a virtue to being a property of the setup.

What I expected wrongly#

An assessment that only lists hits is advertising. Two expectations didn’t hold.

The most expensive misjudgment: that writing things down is enough. For months I put work into ever better instruction files before I honestly drew up the chronicle of these attempts – written rules remain recommendations, no matter how well they are phrased. From this disillusionment came the most important insight of the series: rules that are supposed to apply need a path that enforces them – check before effect, approvals, evidence. If I could take a single sentence away from these months, it would be this one.

The second misjudgment is more pleasant: I had expected the choice of model to become more important over time – that at some point one model would pull ahead and the question “which one?” would turn into a strategic question after all. The opposite happened. For my tasks, several model changes since the opening felt like quality improvements, and with every jump the choice became less important, not more. My way of working hardly noticed the changes. What does make itself felt every time, on the other hand: a badly cut task, a missing piece of context, a rule that only exists in prose. My own share dominates the result – that was there early as a thesis, and by now it concretely decides whether a run holds up: in how the task is cut, in the context provided and in the rules enforced in the path.

What remains open#

Three questions remain unanswered in this interim review.

The junior question. The opening contained a suspicion: the lever works because three decades of judgment stand behind it – and whether it works the same way for someone without that foundation, I didn’t know. I still don’t know today, and the critical installment sharpened the suspicion rather than dispelled it: if you get working code before the pain sets in from which understanding arises, you may never build up the substance that checking draws on. A suspicion is still not knowledge. But it is the question I consider more important than any model question.

Homogenization. That teams asking the same model build more similar solutions is something I have recognized several times since that one observation in February, but never measured. It remains a concern without evidence, and I deliberately leave it at that instead of inflating it into a thesis.

Unlearning my own craft. The house rules from February hold up about as well as New Year’s resolutions: better than nothing, worse than claimed. The deliberate run without AI does happen, but less often than I had planned. The pull of the convenient option is real, every day. Exactly this experience made me skeptical, for the second half of the series, of everything that relies on discipline – and receptive to everything that moves diligence into the setup.

The arc, in hindsight#

The triad of I, We, System was already there as an intention in the opening. What I hadn’t planned is the mechanism that emerged while writing: each level inherits the unsolved problem of the previous one. The discipline of the individual doesn’t scale to the team, so the team writes down its rules; written rules bind nobody, so you need knowledge with owners and a place that survives the session; curated knowledge doesn’t yet answer the question of what is binding, so the hard rules move into the path of a dedicated agent; and so that this doesn’t remain a drawing-board design, you need concepts, methods and processes that keep the beginning small.

flowchart LR
    subgraph AKT1["Act 1 · I"]
        S["Session<br/>generating gets cheap"] --> P["Way of working<br/>checking becomes the bottleneck"]
    end
    subgraph AKT2["Act 2 · We"]
        P --> V["Team variance<br/>discipline doesn’t scale"] --> D["Instruction files<br/>recommendation instead of rule"]
    end
    subgraph AKT3["Act 3 · System"]
        D --> W["Curated knowledge<br/>with owners"] --> R["Rules in the path<br/>agent per domain"] --> B["System architecture<br/>UI, contracts, approvals"]
    end

The sentence that holds this arc together came up in the installment on curated knowledge: it’s not the model that gets smarter – the organization curates its knowledge.

Today’s system view#

At the center is the property this series has been about from the start: the separation of proposal and effect, built as a dedicated approval boundary, not as a blanket switch. A workflow engine can hold back external writes until confirmation. A publishing system can tie the promotion of a preview to a human. A knowledge system can leave changes standing as a proposal until a human decides. An order control system can tie execution to approved orders. The same principle, cut differently each time – because the question of what “taking effect” means has to be answered differently in every domain. Published content takes effect differently from an approved knowledge revision, and both differently from a workflow that writes into a third-party system.

With that, the question the opening asked has shifted too. It was: what can the models do? It is now: what does an organization make of it? Access to an external model API is not the part of this architecture that the organization permanently controls itself. A UI backed by a cleanly cut domain, machine contracts that other systems can rely on, and approval paths that prevent a proposal from taking effect without the intended human decision: an organization has to build these itself, and that is exactly the share that remains its own, independent of any single model.

Position instead of prediction#

I have no basis for judging model capabilities a year from now, productivity curves of entire industries or labor markets, and above all these months have taught me how quickly such statements age. That’s why I’m not formulating a vision, and I explicitly mark what this interim review is: my limited state of experience, not a prediction about inevitable market or organizational developments.

What I can stand behind, because it has been confirmed repeatedly over these months: the lasting value lies less in the individual model than in your own knowledge, enforceable rules, clear domains and verifiable processes. That is not a statement about the future but a description of what remained after every model change – and of what was missing every time a result didn’t hold up.

And for my own desk, one expectation remains that you can measure me against: the first domain agent that has been in operation long enough for its rejections to tell stories – correct noes, overly tight noes, special cases that deserve a verb of their own. Only then will it show whether the cycle from the last installment holds up or just sounds good.

Interim review, not a final word#

This series pauses here, for a simple reason: the supply of observations is used up. What would come now would be extrapolation, and there is enough of that. What this interim review lacks is time in operation. When enough new material has come together again, when the noes of the first domain agent have something to tell, I’ll keep writing. Until then, the resolution from the opening applies unchanged: distrust your own enthusiasm, measure against real tasks, check every result against your own craft. Looking closely pays off, and it doesn’t stop.

Further reading#

Revised and translated from German with the help of AI.