Skip to content

What I Am Critical Of

Four observations and one open hypothesis after intense months with AI – from a possible loss of skills to the worry about more uniform solutions, and my practical countermeasures.

≈ 9 min readSoftwareAIMethodology

On a Thursday evening in January, I almost waved through a pull request that I had commissioned myself. Claude Code had written a database migration, cleanly formatted, with comments, with a rollback path. I read it, nodded, already had the mouse pointer over the merge button – and then, more out of habit than mistrust, I took a look at the affected table in the staging system after all. The migration would have worked. It would just, on the side, have renamed a column that a report accesses, a report the session could not have known about. No drama, quickly found. But the moment before still occupies me today: I had not checked. I had read it the way you read a well-written text – benevolently, going with the flow.

Part 4 of the series “From Session to System” – the series opener is here.

In the last installment, I described why delegating to AI only works for me with a good harness – with tools, feedback and guardrails around the model. At the end there was an unease that has grown louder with every delegated task. Today I want to take this unease seriously. Not as an alarm – I keep working with these tools every day, gladly and productively. But as what criticism should actually be: looking closely before you carry on. In doing so, I separate four observations from one open hypothesis.

If you only review, you unlearn building#

When I rarely write code myself, I now have to distinguish more carefully: am I only missing an API signature, or am I missing practice in the routine? If you need a rarely used library function, you look up its signature – that has always been the case and is not a problem. The problem would be the direction of the movement: if it is no longer just the exception that gets looked up but the routine disappears altogether, the repetition from which fluency arises is missing. Whether and how fast this happens to me, I cannot measure. That is why I deliberately do certain routine tasks without AI and pay attention to where I lack practice.

The CHI 2025 study by Microsoft Research and Carnegie Mellon surveyed 319 knowledge workers on this. In the self-reports, higher confidence in generative AI was associated with less critical thinking being applied. That is a correlation in a survey, not evidence that skills have already eroded. My worry goes one step further than the study’s result: if you hand off routine work and only deal with the exceptions, you could lose exactly the practice you need for the exceptions.

For me, this is a creeping risk that I can actively manage. What concerns me more is what the math looks like for people starting their careers. I learned my craft by falling flat on my face often enough: hunting memory leaks myself, building race conditions myself and understanding them myself. If, during the learning phase, you mostly review generated code, you may get fewer opportunities to build up this substance – that is a guess, not a measurement, and the survey cited says nothing about it. Concretely, what would then be missing is the experience that makes good reviewing possible in the first place: being able to judge whether a timeout, a lock or an error handling approach fits your own system, because you have made such decisions yourself and borne their consequences.

Plausible is not correct#

The scene from the beginning has a name: automation bias. A systematic review by Goddard, Roudsari and Wyatt describes it as the tendency to follow automated recommendations and overlook contradictory cues – the research on it is older than the current language models. In my migration, the clean formatting, the comments and the included rollback path amplified exactly this impression of trustworthiness: I almost approved what I had not checked.

Whether review and the four-eyes principle work systematically worse against such plausible output than against visibly sloppy code, I cannot prove.

In line with this, Perry and colleagues showed in a controlled study that participants with an AI assistant wrote insecure code more often and at the same time were more convinced that they had written secure code. The study does not examine code reviews. For my review practice, the combination is therefore a warning sign, not evidence of effectiveness.

“That’s how the AI did it” is not an answer#

In a review, “That’s what the AI suggested to me” is not enough as a justification for me. The sentence explains the origin, but not the decision. Why is the timeout set to thirty seconds here? Why is that right for our system? If no answer follows, the decision is not justified in the review – because the session that chose the value ended long ago and took its reasons, if there were any, with it.

In a review, there is no third party. There is the person who submits the code and the person who approves it – both with names, both reachable, both still around tomorrow. What the tool suggested is as relevant as the question of which editor the code was typed in. If you open a pull request, you are saying: I stand behind this. If this commitment dissolves because “the AI” is in the room as a diffuse co-author, the review loses its core. Then two people review something nobody feels responsible for.

Debt that nobody took on#

Technical debt can arise from deliberate shortcuts: we build the quick solution now and clean up later. You knew the shortcuts you took deliberately and their consequences. Next to that, I see a possible second variant: code that works, passes all tests – and that nobody on the team really understands. Not because it is incomprehensible, but because nobody ever had to understand it. It was generated, it ran, it was merged.

If a team only roughly follows a generated mechanism when it is built in, the next change may require reconstructing assumptions and the reasons behind decisions before the mechanism can be changed safely. Then technical debt may have arisen without anyone having taken it on deliberately – no note, no decision, no “we’ll clean up later.” Such debt can arise unnoticed because no deliberate shortcut was documented. A system that consists to a larger extent of such code is not broken. It is something worse: not understood while fully functional.

When everyone asks the same model, designs can become more similar#

The last worry is an open hypothesis. A thought experiment makes it tangible: four groups get the same open-ended design task, and all four ask their first question to the same model. How different do the results turn out? If everyone starts with a very similar first answer, that answer can shape the rest of the design. The effect has not been measured and depends, among other things, on the prompt, context, sampling and model. The Microsoft study mentioned does not examine it.

You can shrug and say: converging on proven patterns is a good thing. That is partly true – I am the last person to find an unusual solution good just because it is unusual. But architectural diversity is not an end in itself; it is a search process. Different designs make alternative assumptions and trade-offs visible. My worry is therefore: if many design decisions run through the same filter, the space of seriously considered alternatives narrows. I cannot measure that.

Criticism does not mean rejection#

At this point, the text could tip over – into cultural pessimism, into “back then we still did real programming.” That would be convenient and wrong. None of these points is an argument against the tools. Each one is an argument against a particular way of using them: the passive one. Skills do not decay because AI exists but because I stop practicing. Automation bias strikes because I confuse plausibility with correctness. Responsibility diffuses because I let it diffuse. That is the catch: the answer does not lie in the next model release. It lies with me.

I have set up a few rules for myself. No words of wisdom, more like house rules:

  • Regularly work deliberately without AI. Not a ritual of renunciation, more like a run without a watch: certain tasks – an algorithm, a data model, a tricky bug fix – by hand, to feel where it snags. Where it snags, I check whether I lack practice or whether I just do not have a rarely used API at hand.
  • An obligation to understand before the merge. Nothing goes into the main branch that I could not explain to a colleague at the whiteboard – line by line for critical spots. Critical spots are, for example, possible data loss, permission checks, concurrency and public APIs. “Roughly followed” does not count.
  • My own judgment first, then the model. For design questions, I write down my position in three sentences before I ask. It does not have to be the better one; it is an anchor against the plausible first answer – deviations then become real information.
  • Name the origin, keep the responsibility. When code was generated, I say so openly in the review – and take it on anyway as if I had typed it.

None of this is original. The extra review costs me time, but it makes the approval more traceable. And I notice: it takes effort. In everyday work, it is easier to skip it.

But this is exactly where the reach of this text ends. Everything I have described here are measures of an individual – my rules, my discipline, my house rules. That reaches exactly as far as my self-control does, and on bad days it reaches surprisingly little. A team cannot rely on the private discipline of its members; ten people with ten different sets of house rules do not yet make a shared way of working. What happens to these observations when not one person works like this but an entire team – with different experience, different levels of care and deadline pressure on top – that is the question I have set myself for the next installment.

Further reading#

Revised and translated from German with the help of AI.