My Success Cannot Be Copied
A colleague takes over my AI setup and gets different results. Tools, tests and conventions can be transferred; how you cut tasks, how you review and the course of an individual session only to a limited degree.
In February, a colleague – an experienced developer with good judgment, no beginner – asked me whether he could have my setup. He had watched me work with Claude Code for weeks and wanted the same: the same tools, the same configuration, the same results. I gave him everything. The complete toolchain. My instruction files, grown over months. My prompt collection for recurring tasks, my checklists for reviews, even my notes on which tasks I delegate and which I do not. There was no secret remainder. He really had everything I have.
Part 5 of the series “From Session to System” – start here with the opener.
Two weeks later, we sat down for coffee, and he was disillusioned. Not frustrated in the sense of “the tool is useless” – he had had some genuinely useful moments. But what he had seen with me, the matter-of-factness with which tasks go out and checked results come back, did not happen for him. For him, the AI went in the wrong direction three times, built things he did not want, and he spent more time cleaning up than he used to spend writing the code himself. His sentence stuck with me: “I have your setup. Why don’t I have your results?”
The question occupied me longer than any tool problem of the past months. Because the obvious answers are all wrong. It was not the model – we used the same one. Not the configuration – it was identical, file for file. Not a lack of experience as a developer – he is at least as good as I am. What he was missing was not in any of the files I had given him. And that is exactly the point where this series has to make the leap: from me to the team. So far, I have described why the harness matters more than the model and what makes me uneasy about my own way of working. Both were the perspective of an individual. As soon as a second person comes into play, a problem shows up that I had previously only suspected: variance. And on three levels at once.
First level: the same task, two sessions, two results#
The first level has nothing to do with my colleague yet. I can demonstrate it on my own machine.
I give the same AI the same task twice – under largely the same visible conditions: same prompt, same repository, same instruction files. Session one decides to read the tests first, finds an existing pattern there and extends it cleanly. Session two jumps straight into the production code, considers a helper function unused, removes it “while it is at it” and delivers a solution that runs but is cut differently from everything around it. Both sessions got the same assignment. Both report confidently that they are done. The results differ in structure, style and side effects.
In my runs, even early decisions differed – for example which file was read first or which pattern served as the model. After that, the subsequent steps diverged too.
For me alone, that is manageable. I have learned to check results instead of prescribing paths, and in episode 3 I described how much of this the harness catches: tools, feedback loops, check steps. But even here, the mistake of confusing “best practices” with reproducible results becomes apparent: the same success case cannot be repeated exactly. When I show my colleague how great a session went yesterday, I am showing him a single run – even for me, the next one would look different.
Second level: the knowledge that is in no file#
The second level is the one where my colleague actually failed. And it is more uncomfortable, because it disguises itself as a tool question but is a question of experience.
When we talked through his disillusioning two weeks, I noticed: he did the same things I do – just in different places. He delegated, but the wrong tasks. He checked, but looked at the wrong things. He stopped, but too late. Three examples from our conversation:
- He gave the AI a task I would never delegate as a whole – a change across three modules with unclear domain logic – and was surprised by the chaos. I would have cut it into three checkable chunks beforehand. This cut is not in any of my prompts; I make it in my head before I even start typing.
- He read the AI’s explanations and found them convincing. I hardly read the explanations anymore – I look at the diff, the tests and the places where, in my experience, unwanted side effects occur. Where you look is judgment, not configuration.
- He had a session make three rounds of fixes before he gave up. I stop after the first failed fix attempt and rephrase the task, because I have learned: once the direction is wrong, steering back costs more than starting over. That is not written down anywhere either – this stopping rule comes from repeated failed attempts.
How you cut tasks, where you look when reviewing, when you stop – that is tacit knowledge in the classic sense. You acquire it through friction, not through reading. My instruction files are the sediment of this knowledge, but they are not the knowledge itself. They tell the AI what to do. They do not tell my colleague what he should do while the AI is working.
The METR study from summer 2025 made a related effect measurable: for experienced open source developers, tasks with early-2025 AI tools took 19 percent longer on average – even though the participants expected to save time. You can argue about how far the study generalizes; the conditions were special. But the documented finding fits what I saw with my colleague: expected time savings and measured working time can diverge.
Third level: there is no identical run#
The third level is often overlooked because it sounds so banal: my colleague works on different tasks than I do.
Not fundamentally different – same company, related systems, a similar stack. But his specific task on a specific Tuesday is never my specific task. His module has a different history, different legacy, different unwritten conventions. His bugs sit in different places. Even if he had my judgment and the session lottery went his way: he still could not reproduce my success, because the experimental setup “same task” simply does not exist in real work.
That fundamentally distinguishes AI tools from almost everything we have introduced in teams before. Formatters and CI pipelines aim for reproducible execution under the same version, configuration and environment; a framework API provides the same contract. With these tools, the rule is: one person finds the good configuration, everyone adopts it, everyone gets the benefit. This exact pattern is what we all have in mind when we say “share the setup” – and this exact pattern does not apply here. The benefit of an AI session comes from the interplay of task, context, person and chance. Three of these four factors are different for every team member.
What organizations regularly conclude wrongly from this#
At the organizational level, these differences can lead to false productivity assumptions. Because the individual observation “it works for Mike” invites three fallacies.
The first fallacy: license bought means team productive. An enterprise procures access for all developers, books it as “AI adoption” and expects the productivity leap that the power users showed in the pilot project. But the power users were fast precisely because they had built up tacit knowledge over months. The license buys the tool, not the second level. The 2025 DORA report puts it in a formula I have been quoting constantly ever since: AI amplifies what is already there. Teams with good feedback loops and clean architecture get better; teams with weak processes get their weaknesses back amplified.
The second fallacy: sharing a prompt library is a rollout. That was my own mistake with my colleague, just on a small scale. Prompts and instruction files are sediment – useful, but dead without the practice they came from. A shared prompt collection without shared judgment produces people who read convincing explanations instead of checking diffs. The 2025 Stack Overflow survey shows the consequences at scale: AI tools are widely used, but trust remains limited, and a frequently cited frustration is code that is “almost right.” Recognizing almost right is exactly the ability that cannot be distributed by file alone.
The third fallacy: what works in the pilot scales linearly. It does not, because of all three levels of variance at once. A pilot can include motivated power users (person variance), selected tasks that are easy to cut (task variance) and selectively reported success stories – failed sessions may be missing from the management presentation (session variance). If you build a projection for 200 developers from that, you may be calculating with an unrepresentative average.
I put it pointedly because I already used the sentence in episode 4 and here it describes the organizational consequence: the success stories of individuals are real, and they are still not a blueprint. Tools, tests, conventions and acceptance criteria can be copied. My concrete result also arises from task, context, review judgment and the course of the session. No setup file transfers that part.
Variance is also a source of learning#
This sounds bleaker than I mean it. Because the same variance that prevents copying is a useful diagnostic source for a team.
When my colleague and I dissected his two weeks, something remarkable happened: for the first time, I had to say out loud why I cut tasks the way I do. Why I ignore explanations and read diffs. Why I stop after one failed fix attempt. That was tacit knowledge that I had not perceived as knowledge myself – for me, it was simply “how you do it.” Only the difference between his way of working and mine made it visible. Exactly where two people do the same thing differently, the tacit knowledge lies in plain sight. You just have to look instead of dismissing the difference as a failure of the slower one.
My conclusion after that February: if individual experiences are to flow together in a team, the prompts are not the valuable part. The differences are. Two developers, the same class of task, different results – that is not an annoyance; it is a finding. Each of these differences points to an unspoken rule, a heuristic, a piece of judgment that so far existed only in one head.
By the way, my colleague still works with the setup – now with his own instruction files that differ more and more from mine, and with much better results. Not because he copies me better now, but because he stopped trying. What he took over from me is not the files. It is the questions he asks himself before every delegation.
And that brings me to the consequence that every team sets out to draw at this point, mine included: if the decisive knowledge is tacit, then we simply make it explicit. We sit down, spell out the rules that so far existed only in heads and write them down – in exactly the files the AI reads in every session. Task cutting, review rules, stopping criteria, all of it goes in. That has to be enough. Why it is not – and why the AI follows our carefully written rules surprisingly selectively – is what the next episode is about.
Further reading#
Revised and translated from German with the help of AI.