Clean Grounding I: Why AI Guesses So Convincingly
Why language models invent plausible things, why more training doesn’t reliably eliminate the problem and what grounding has to do with it.
A few weeks ago I was sitting in a review of generated code. The assistant had suggested retry logic for our internal HTTP client – a client that exists only at our company, a thin TypeScript layer over fetch with a few in-house conventions. The suggestion: client.withRetry({ backoff: "exponential", maxAttempts: 3 }). Cleanly named, in the style of our codebase, with a comment on why exponential backoff was the right choice here. Plus a short explanation in the chat of how the method relates to our timeout handling.
The method doesn’t exist. It never existed. Our client configures retries in a completely different place, when the instance is created, not at the call. But the frightening part wasn’t the mistake. I make mistakes too. The frightening part was the quality of the invention: the name fit our conventions perfectly, the signature looked like one of ours, the reasoning was sound from a domain point of view. If a human had suggested the method, I would have said: this person really knows our code well. Except that the suggestion was a complete guess – and was delivered with the same confidence as the twenty correct suggestions before it.
Part 7 of the series “From Session to System” – the series opener is here.
If you work seriously with Copilot, Claude Code or Cursor, you can run into something like this. At first, I looked at such errors like a bug that should disappear with the next model generation. Today I don’t rely on that. Better models can reduce hallucinations, but they don’t reliably eliminate them – and since that became clear to me, I work differently with these systems.
The model doesn’t reliably know the limit of its knowledge#
To understand why the invention sounded so convincing, I had to remind myself what a language model actually does: it continues text. From billions of examples it has learned which continuation is likely after a given beginning. That is not a disparaging simplification but the mechanism – and it is astonishingly powerful, as I described in part 2 of this series. But it has a property that can’t be argued away: true and false statements both arise as likely continuations; a truth check doesn’t automatically follow from this mechanism.
A human developer – fallible as they are – has a sense of whether they know something or are guessing. If I ask a colleague about a method in our internal library, they answer either “it’s called this” or “I’d have to look it up”. This having-to-look-it-up is metacognition: knowledge about your own knowledge, a sense of your own uncertainty. Exactly this faculty is missing in the model. It has no inner state that distinguishes between “this appeared a thousand times in the training corpus” and “this sounds the way it ought to sound”.
That’s why the confidence that irritates us so much is not arrogance but an artifact. The tone of a text is part of the continuation. Technical documentation, Stack Overflow answers, code comments – the material the model learned from is overwhelmingly written in a tone of certainty. Nobody documents an API with “I think this method exists”. So the invented method sounds like documentation too. The confident tone is not a signal of correctness. It never was.
Two papers support parts of this explanation. In 2024, Ziwei Xu and colleagues argued formally that hallucinations cannot be completely eliminated for large language models. And in a publication of its own – available online as of August 2026 – OpenAI describes a second mechanism: training and evaluation incentives can push models toward guessing instead of admitting uncertainty. A model that says “I don’t know” when uncertain can score worse on benchmarks that reward correct answers but don’t properly account for abstentions than one that guesses confidently – just as a multiple-choice test teaches students to tick a box at random when they don’t know instead of leaving the field empty.
Symbols that refer to symbols#
There is an even deeper level, and it is older than the current models. In 1990 – long before today’s large language models – the cognitive scientist Stevan Harnad formulated the Symbol Grounding Problem. His question: how does meaning get into a system that only manipulates symbols according to formal rules? His image: try to learn Chinese exclusively from a Chinese-Chinese dictionary. Every character is explained by other characters, and those in turn by more characters – you go around in circles without ever arriving at the thing itself. Symbols refer to symbols; nowhere does the system touch the world.
I usually hesitate to drag cognitive science papers into everyday debates, because that often creates more fog than clarity. But here the analogy fits uncomfortably well. A language model knows text about the world, not the world. It has read what people have written about HTTP clients, about retries, about exponential backoff – and has learned statistical relationships between tokens from it. In any case, the answer in my opening scene gave no hint of a check against the thing itself: no compiler rejecting the call, no test turning red, no production alerting at three in the morning. Where I as a developer have calibrated my concepts against experience, the model has calibrated its concepts against other concepts. The dictionary explains itself, and it explains itself fluently.
You don’t have to adopt Harnad’s problem in its full philosophical sharpness to see the practical core: a system whose entire “knowledge” consists of text statistics can’t tell a plausible invention from a fact, because both are made of the same stuff. For the model, withRetry was just as real as any method that actually exists – a likely symbol among other symbols.
Why bigger models mitigate the problem but don’t solve it#
The objection is obvious: the models have gotten better, haven’t they? In my work, at any rate, I now rarely catch current models making things up freely when it comes to public, well-documented knowledge.
But “more rarely” only means “less often”, not “fundamentally different”. The mechanism stays the same: a bigger model has learned finer statistical relationships – it guesses less often because it has seen more, not because it has stopped guessing. At the point where its material ends, it still produces the most likely continuation, in the same confident tone. The line between knowledge and guesswork moves outward, but it doesn’t disappear, and the model still doesn’t show it to me. That is exactly what makes the remaining cases so treacherous: the rarer the error, the less I expect it – and the more unchecked the suggestion goes through.
Without internal sources, organization-specific facts remain guesswork#
And that brings me to the point that occupies me most in my daily work. Everything said so far applies to public knowledge – React, PostgreSQL, the AWS APIs. There I rarely see the model guess – the public material is huge. But the moment I open the door to a specific organization, the ratio flips. Because what counts there isn’t reliably accessible to the model without provided internal sources:
- the internal libraries and services with their actual signatures and quirks
- the in-house conventions – how errors are handled, how modules are split, what a deployment has to go through
- the terms of the domain that mean something precise inside the company and something else or nothing at all outside it
The case from the opening scene showed me how far this goes. The model had never seen our HTTP client, so it invented it – and did so in the way such a client typically looks. That’s why the suggestion was so good: it was the ideal form of an internal HTTP client, distilled from public patterns. Just not ours. Without an internal source, the actual signatures couldn’t reliably be derived by the model; the invention was an obvious move and looked deceptively similar to real knowledge, because the model is so good at continuing the style of a codebase.
In hindsight, this also explains why my attempts with instruction files went so sobering – I told that story in detail in the last part. Back then I still thought the problem was compliance: the AI doesn’t stick to our rules, so the rules have to be stated more clearly, more strictly, more prominently. But even if an AGENTS.md were followed perfectly, it answers the wrong question. An instruction tells the system how to behave. It can’t give it what it actually lacks: the facts. “Don’t invent methods” is a helpless instruction to a system that doesn’t notice when it invents. The model doesn’t lack the will; it lacks the material. I was working on the motivation of a system that doesn’t have a motivation problem but a knowledge problem.
Grounding: providing internal facts in the working context#
For this answer to the knowledge problem, I use the term grounding here – anchoring answers in verifiable facts. At its core, the idea is simple. If the model can’t know my reality from training, then I have to give it that reality at the time of the work: the real signature instead of the likely one, the actual schema instead of the typical one, our convention instead of the industry standard. Not as a rule of behavior, but as material. The model should be able to look things up instead of having to guess – roughly the way a good IDE with type information keeps me from typing method names from memory.
Here I use grounding for providing concrete source material in the working context, and I thereby distinguish it from mere instructions that formulate rules of behavior. Grounding is not a product and not a feature you switch on. It is first of all an insight about the division of labor: the model provides language understanding, pattern recognition and synthesis – the facts about my world have to come from my world itself. And, to be honest, it isn’t a complete solution either. A grounded model can still connect the facts it was given incorrectly, and if you have ever tried to dump “simply all the documents” into the context, you know that this brings new problems. The difference lies elsewhere: with a grounded system, I argue about the quality of my sources. Without sources, I check free additions drawn from general patterns.
With that, the conclusion of this part is settled, even if it saves no one any work: better models and stricter instructions don’t reliably eliminate convincing wrong answers. Grounding provides concrete source material where the system would otherwise have to fill in from general patterns. Which is where the real work begins – because “provide internal facts” is quickly said. Which facts, exactly, in what form, at which point in the workflow? That is not a rhetorical question but the next part.
Further reading#
- Stevan Harnad: The Symbol Grounding Problem (Physica D, 1990) – the original paper with the dictionary argument
- Ziwei Xu et al.: Hallucination is Inevitable – An Innate Limitation of Large Language Models (2024) – the formal argument that hallucination cannot be eliminated for this class of models
- OpenAI: Why Language Models Hallucinate – why training and evaluation incentives can reward guessing (retrieved in August 2026)
- Lilian Weng: Extrinsic Hallucinations in LLMs (2024) – a detailed overview of causes, measurement and countermeasures
Revised and translated from German with the help of AI.