Skip to content

Clean Grounding II: Grounding Answers

My working model for grounding distinguishes conventions, documents and API descriptions. What stays in the scarce context, I prioritize by task and by the cost of fetching it again.

≈ 11 min readSoftwareAIMethodologyDelivery

A few weeks ago, a colleague and I sat in front of a half-finished piece of integration. The AI was supposed to write a client for an internal API, and it did what it always does: it suggested customerId. In our system, the field is called customer_ref – for historical reasons that no one defends anymore but everyone knows. We had already explained that to the assistant twice that week. This time I did something different: I didn’t contradict it; I put the OpenAPI description of the API into the context. The next suggestion was right. Not roughly, but exactly – including the somewhat idiosyncratic pagination that no model in the world could have made up, because it was simply a decision of ours. Same model, same day, same task. The difference: it no longer had to guess.

Part 8 of the series “From Session to System” – the opening post is here.

In the last episode, I described why AI guesses so convincingly: a language model uses symbols to refer to symbols, not to my systems. It knows a thousand APIs that look similar to mine – and that is exactly why it invents plausibly. That was the diagnosis. This episode is the constructive side: how do I give the AI my reality? For myself, I call this clean grounding – grounding answers instead of leaving them hanging in the air. And as so often, the core is less a tool than a sorting: what knowledge is there at all, where does it come from, and when does it belong in the context?

My working model: three layers of knowledge#

When I look at what gives my answers solid ground in practice, I keep ending up with three layers. This is my sorting for this work, not a general taxonomy of knowledge. The layers differ not only in content but in their character – and that is the real point.

flowchart TB
    subgraph Schichten["Three layers of knowledge"]
        K["Conventions<br/>how we work<br/>prescriptive, curated"]
        D["Documents<br/>runbooks, specs, decisions<br/>descriptive, aging"]
        S["API descriptions<br/>OpenAPI, GraphQL introspection, schemas<br/>machine-checkable"]
    end
    K --> E["Grounding while building<br/>and preparing"]
    D --> E
    S --> E
    E --> A["Checked work result<br/>document, plan, code, configured flow"]

The first layer is our conventions and ways of working. This is prescriptive knowledge: not “this is how the world is,” but “this is how we do it here.” Error handling belongs to the caller, migrations never run automatically, commit messages explain the why, reviews check the interface first. None of this is in an API description, and none of it can be guessed by a model – these are decisions, not facts. In episode 6, I told how I tried to pack exactly this knowledge into instruction files like AGENTS.md or CLAUDE.md – and where recommendations reach their limits. Giving up was not the lesson. This layer has to be small, dense and curated: a short list of rules the team stands behind can be followed and enforced in review. A long collection of hints competes with itself for attention.

The second layer is our documents: runbooks, specifications, architecture decisions, operations notes. This is descriptive knowledge – it describes what applies here and why it came about this way. A recorded architecture decision can prevent the AI from overriding three-year-old system boundaries with its suggestion. A runbook explains what really needs to be done for a particular alert, instead of what sounds generically sensible. But I’ve learned to approach this layer with a certain skepticism: documents age. They describe the state of back then, not of today. A runbook that no one has touched in two years can mislead the AI just as much as its own guessing – only then the answer sounds even more convincing because it cites “our docs.”

The third layer is real API descriptions – and it’s my favorite, because it has a character the other two lack: tools can read their structure, even if schema and runtime behavior can drift apart. An OpenAPI description, a GraphQL schema queried via introspection, a JSON Schema, a database schema – that is not a colleague’s memory of the system but a machine-readable description of structure or API. Introspection shows the schema this environment offers at this point in time: if the field is called customer_ref, that’s what it says there. That proves neither correct resolvers nor agreement with other deployments or with data-dependent responses. Depending on the artifact, I can read field names, types or endpoints from it instead of guessing them from conventions or documents.

The three layers answer different questions, and they don’t replace each other:

  • Conventions say how something should be built – style, boundaries, responsibilities.
  • Documents say why things are the way they are – context and history.
  • The respective API description names fields, types or endpoints. Whether the implementation matches this state and works has to be checked separately.

If you supply only the third layer, you get code that matches the described contract but doesn’t necessarily work at runtime. If you supply only the first, you get stylistically clean code against an invented API. Only together do they ground an answer.

Look it up instead of guessing#

The principle behind all three layers is the same, and it’s so unspectacular that it’s easy to read past: look it up instead of guessing. The easiest way to understand it is through an analogy everyone knows from the editor: IntelliSense.

When I type customer. in my editor, the editor doesn’t guess which fields a customer probably has. It looks them up. A type-based editor derives suggestions from the available type definition, not from statistical similarity. Exactly this derivation is what a language model lacks when we let it work without grounding.

An agent with suitable tools and read permissions can look things up. It can read an OpenAPI file, query a GraphQL schema via introspection, open a type definition in the repository. The Model Context Protocol provides tools and context sources for this through a protocol contract. And such an agent can go one step further, beyond IntelliSense: it can try endpoints. A read-only call against the real API, a look at the actual response structure, and the next suggestion is based on observed data instead of invention. “This field probably exists” becomes “I have seen this field – in this response, with these permissions, in this environment and at this point in time.” That is a categorical difference, not a gradual one, even if a single response doesn’t prove the behavior for other data, permissions, environments and points in time.

Exploratory read access must be authorized; whether it is free of side effects in domain terms has to be clarified beforehand. Such calls are valuable for research. Writing calls “just to try it out” are a topic of their own with their own safeguards – I’ll come to that later in this series. For grounding answers, the reading side is almost always enough.

Grounding of repeatable workflows belongs at build time#

The second principle took longer to sink in for me, and it’s the one with the biggest consequences: when does grounding happen? Wherever a repeatable workflow has an artifact that can be checked in advance, grounding belongs at build and preparation time – not in the critical path of execution. If, on the other hand, the question only arises at runtime, the grounding has to happen there too.

The diagram above shows this case for an artifact that can be checked in advance.

At this point I have to place RAG, because it’s the first word in the room as soon as someone says “our own knowledge” and “AI” in the same sentence. Retrieval-Augmented Generation – described in 2020 by Lewis and colleagues – means: at query time, matching text passages are searched in a corpus and passed to the model. That is exactly right when the question only arises at runtime and the answer has to come from a body of documents – an assistant that answers questions about the handbook, a search across internal notes, support cases. Such cases correspond to the pattern of retrieval and generation described in the RAG paper.

It goes wrong when RAG becomes the reflex for everything. Because retrieval at runtime has two properties that hurt in day-to-day work: additional latency and possible changes in results when the index, ranking or search method changes. Every execution searches anew, and what gets found depends on embeddings, rankings and the state of the index – the same workflow can see different text passages today than yesterday and therefore behave differently. For an exploratory question, that’s tolerable. For a workflow that runs every night and whose result people rely on, changing hits can lead to non-reproducible results.

That is why, for such repeatable workflows, I pull grounding forward. The knowledge – the conventions, the documents, the API descriptions – flows in while something is being created: a document, a plan, a piece of code, a configured workflow. This work result is checked, by me or by a colleague, and from then on it is fixed. The later execution doesn’t roll the dice again; it executes what was checked. The difference is the same as between a program that guesses its dependencies from the network on every start and a build with pinned versions: all the uncertainty belongs in the fleeting, creative moment – it doesn’t belong in what then runs again every night.

This is not a rejection of RAG but a clarification of responsibilities. Search across documents at runtime, where the question arises at runtime. Grounding at build time, where the result is an artifact that is meant to last. In my work, the second case is the more common one – yet it is still discussed in the vocabulary of the first.

Context economics: what goes last#

The third principle sounds trivial and decides a surprising amount in practice: context is scarce. Yes, the windows have gotten big. But if you work with agent sessions for a while, you know that “big” doesn’t mean “inexhaustible” – long sessions, files read, tool outputs and intermediate states fill every window, and at some point things get trimmed, summarized, left out. Anthropic put this in a nutshell under the term context engineering: context is a finite resource; relevant information is found deliberately and loaded only when needed. A large, unfiltered context is not a mark of quality – the source that is authoritative for the task belongs in it at the time of the work.

For my grounding, this leads to a prioritization rule: when something has to leave the context under pressure, I first remove material that can be looked up again in a targeted way. A long runbook stays in the context only with the paragraphs that are relevant right now. Small, task-relevant conventions I hold on to longer, because their loss is harder to notice. For a different task, this order can look different; what matters is relevance and the cost of fetching it again, not the name of the layer.

That’s why I keep convention documents small and tailored to the task. A sprawling document competes with sources and tool outputs for space. Curating here means deciding deliberately what has to stay in the context for the next step.

What grounding doesn’t do#

Honesty includes the counter-calculation, and I don’t want to squeeze it in at the end as if it were a footnote. Grounding doesn’t eliminate guessing. With the OpenAPI description loaded, the next suggestion contained no invented field names or structures. But a language model remains a language model: it can have a correct specification in the context and still draw a conclusion from it that doesn’t hold. It can read two documents together that contradict each other and elegantly bridge the contradiction instead of reporting it. And it can be grounded in an aged runbook – then the answer is cleanly reasoned and still wrong, only harder to recognize as wrong.

So checking remains mandatory. The grounded work result is a better candidate, not a final verdict. What has changed for me is the kind of checking: I search less often for freely invented fields and more often for wrong conclusions from correct sources. That’s a better problem – the same kind of problem I would have with a human colleague who had conscientiously read up. But it is a problem, and grounding replaces neither consistency checks nor tests – without such checks, wrong conclusions can go undetected.

One question hasn’t let go of me since. Everything I’ve described here grounds one session: the conventions, the documents, the interfaces flow into one working context, and at the end there is a checked result. But the session ends. The window closes, the context is gone – and with it everything the AI learned in those two hours about our system, our quirks, our corrections. Tomorrow I’ll explain customer_ref for the third time, or I’ll have found a way for knowledge to survive the session. What that can look like – without accumulating outdated, unfindable or unsupported entries – is the subject of the next episode.

Further reading#

Revised and translated from German with the help of AI.