> ## Content Index > Fetch the complete content index at: https://bastian.ghost.preview.themeanax.com/llms.txt > Use this file to discover other available public pages before exploring further. # What Retrieval Actually Fixes, and What It Doesn't - URL: https://bastian.ghost.preview.themeanax.com/what-retrieval-actually-fixes-and-what-it-doesn-t/ - Published: 2026-07-16T14:12:39.000Z - Updated: 2026-09-07T12:48:17.000Z - Description: We spent a quarter trying to get artificial intelligence right, and the useful lessons were not the ones we expected. - Author: Bastian Team - Tags: #themeseed, Artificial intelligence, Field guide, Artificial Most teams arrive at artificial intelligence the same way: something breaks, and the fix becomes a habit. ## What the data actually shows What looks like a process problem is frequently an ownership problem. If you learn on Friday what you assumed on Monday, the assumption never has time to become an architecture. We ran both approaches in parallel for six weeks. The difference was smaller than the cost of the debate about it. ![robot and human hands reaching toward ai text](https://bastian.ghost.preview.themeanax.com/content/images/2026/09/photo-1694903089438-bf28d4697d9a-2.jpg) Photo by Igor Omilaev on Unsplash Measurement is usually where this falls apart. Success has many causes and teaches very little; failure tends to have one, and it is usually obvious in hindsight. Most of the difficulty lives at the boundaries, not in the middle. Being right sixty per cent of the time builds exactly the kind of confidence that makes the other forty per cent expensive. ## The incentive problem The interesting constraint is almost never the one in the brief. Teams that pick both end up with neither, and usually discover this at the point where reversing would have mattered. Ask what would have to be true for the opposite approach to be correct, and see whether anyone can answer. The RAG Reality Check: When Retrieval-Augmented Generation Works (and When It Fails) It helps to separate the decision from the execution. Where a design is obvious the prose is short, so the length of an explanation is a reasonable proxy for where to look next. One team we spoke to cut their review stage entirely and found throughput unchanged, which told them something the metrics had not. ## How we got here The second-order effects arrive about a quarter after the first-order ones. The things that are easy to count are rarely the things that matter, and once a number reaches a dashboard it starts shaping behaviour whether or not it deserves to. The caveat is that all of this assumes the underlying goal is settled, which is frequently the actual problem. The default answer is right often enough to be dangerous. The stated constraint is usually a proxy for a real one nobody wants to say aloud, and optimising the proxy is wasted effort. The evidence here is thinner than anyone quoting it tends to admit. ## A different reading The expensive mistakes here are rarely the technical ones. The decision is usually cheap and reversible; the execution is where the cost lives, and that is where the argument should have happened. The clearest signal was that people stopped asking where things were. The tooling question is downstream of the constraint question. Cutting scope early is cheap and slightly embarrassing; cutting it late is expensive and deeply embarrassing. It is worth saying that we have not run this long enough to be confident. Scope is the variable everyone adjusts last and should adjust first. The first quarter shows the intended effect; the second shows what the intended effect displaced. Try writing the constraint on one line before opening a vendor comparison; the line is usually harder than the comparison. ## A more modest claim Documentation is a symptom: you write it where the design is unclear. Handoffs between people who each hold a coherent local picture and no shared one produce most of the pain later attributed to tooling. The first thing to establish is what you are actually optimising for. When responsibility is spread across a group, the work that falls between the named parts is the work that does not happen. In practice the answer showed up in the calendar before it showed up in the dashboard. The compounding effects matter far more than the individual wins. They are decisions made quickly, defended slowly, and built upon for six months before anyone recalculates. > The bottleneck is never where you think it is — that is what makes it a bottleneck. — Overheard in a retrospective ## The received wisdom A shared definition of "done" removes more friction than any tool. Choosing infrastructure before agreeing what it is for is how organisations end up maintaining a system nobody wanted. A useful test: if this disappeared tomorrow, how long before anyone noticed? Consider the failure mode rather than the success case. A team that changes approach every quarter pays a coordination tax that routinely exceeds whatever the change was meant to fix. The version of this that works fits on an index card. The version that fails needs an onboarding session. Nobody gets credit for the work that did not need doing. A small improvement applied consistently beats a dramatic one applied once, which is unsatisfying advice precisely because it is correct. Set a date at which you will stop, and write down in advance what would make you stop earlier. A few things worth checking before you commit: - Agree on what "done" means, in writing, before starting - Write the constraint down before choosing a tool - Decide in advance what would make you stop ## The objection worth taking seriously Feedback loops shorter than the planning cycle change everything. Artificial intelligence rewards clarity here more than almost anywhere else, because the wrong target produces work that looks productive and moves nothing. This is easier to write than to hold to when a deadline appears. There is a version of artificial intelligence that is mostly ritual. It is comfortable, it is legible to management, and it is close to worthless once you measure what it actually changes. When we mapped it out, four of the seven steps existed only to compensate for the second one. ## What would change our mind Consistency is worth more than any individual improvement to artificial intelligence. Most disagreements that present as strategic turn out, on inspection, to be two people using one word for two things. Reasonable people land elsewhere on this, usually because their constraints differ more than the vocabulary suggests. Speed and reversibility are the trade-off worth naming out loud. Subtraction is structurally underrated: the meeting that stopped happening leaves no artefact to point at in a review. The version of this that works fits on an index card. The version that fails needs an onboarding session. Documentation is a symptom: you write it where the design is unclear. A small improvement applied consistently beats a dramatic one applied once, which is unsatisfying advice precisely because it is correct. The clearest signal was that people stopped asking where things were. ## Where this leaves us Most of the difficulty lives at the boundaries, not in the middle. The stated constraint is usually a proxy for a real one nobody wants to say aloud, and optimising the proxy is wasted effort. We ran both approaches in parallel for six weeks. The difference was smaller than the cost of the debate about it. The interesting constraint is almost never the one in the brief. Being right sixty per cent of the time builds exactly the kind of confidence that makes the other forty per cent expensive. The counter-argument deserves a hearing, and it is stronger than its usual proponents make it sound. Speed and reversibility are the trade-off worth naming out loud. Subtraction is structurally underrated: the meeting that stopped happening leaves no artefact to point at in a review. A useful test: if this disappeared tomorrow, how long before anyone noticed? ## What the data actually shows The expensive mistakes here are rarely the technical ones. It is comfortable, it is legible to management, and it is close to worthless once you measure what it actually changes. In practice the answer showed up in the calendar before it showed up in the dashboard. Measurement is usually where this falls apart. Success has many causes and teaches very little; failure tends to have one, and it is usually obvious in hindsight. A shared definition of "done" removes more friction than any tool. They are decisions made quickly, defended slowly, and built upon for six months before anyone recalculates. ## The incentive problem Feedback loops shorter than the planning cycle change everything. Artificial intelligence rewards clarity here more than almost anywhere else, because the wrong target produces work that looks productive and moves nothing. Set a date at which you will stop, and write down in advance what would make you stop earlier. The second-order effects arrive about a quarter after the first-order ones. Most disagreements that present as strategic turn out, on inspection, to be two people using one word for two things. ## How we got here Consider the failure mode rather than the success case. When responsibility is spread across a group, the work that falls between the named parts is the work that does not happen. None of this generalises perfectly. Take the parts that map onto your constraints and discard the rest — that is what the framing is for.