Small models, large context, and the return of retrieval
The scaling story quietly forked. One branch kept getting bigger; the other got better at knowing what to read, and it is the second one showing up in production.
For several years the answer to almost every capability question was more parameters, and it was a good answer because it kept being right. What has changed is not that scaling stopped working — it did not — but that a second axis became cheap enough to trade against it, and on a large class of tasks the trade is now lopsided.
The economics of knowing where to look#
A model that has memorised a fact answers instantly and cannot tell you when it learned it, whether it is still true, or where it came from. A smaller model with good retrieval answers slightly slower, cites its source, and updates the moment the source does. For consumer chat the first is usually preferable. For anything with an audit requirement the second is not merely preferable, it is the only admissible option.
| Dimension | Parametric | Retrieval-augmented |
|---|---|---|
| Marginal cost per query | High and fixed | Lower, scales with context |
| Freshness | Training cutoff | Index freshness |
| Provenance | Absent | Structural |
| Failure mode | Confident fabrication | Confident retrieval of the wrong document |
That last row is where practitioners have moved. Both approaches fail; the retrieval failure leaves a trail. When a retrieval system is wrong you can look at what it fetched and see immediately why the answer went where it went, and that debuggability compounds over the life of a system in a way that raw capability does not.
A system that fails legibly beats a slightly better system that fails silently, and it is not close.
What actually got better#
- Chunking stopped being naive. Boundaries follow document structure rather than token counts, which removed a large fraction of the retrieval failures that used to be blamed on embeddings.
- Rerankers got cheap enough to run on every query rather than as an optimisation.
- Long contexts stopped degrading in the middle, which made the retrieve-a-lot-and-let-the-model-sort-it strategy viable.
- Evaluation improved — the single largest practical gain, because most retrieval systems were previously tuned against intuition.
None of these is a breakthrough. All of them are engineering, and the aggregate effect is that a small model with a well-built index now beats a much larger model without one on most tasks that involve a corpus you own. Which is nearly every task inside a company.