We read a lot of papers we never build anything from. That's not a criticism of the research — it's the normal gap between "this improved a benchmark" and "this survives contact with a real product, real users and a real cost budget." The filter for closing that gap is the useful part.
The three questions that filter out most hype
- 1Does the improvement hold outside the paper's own benchmark? A technique tuned to beat one eval often regresses on tasks that eval didn't measure.
- 2What's the actual cost/latency trade-off? A reasoning gain that triples inference cost or latency rarely survives a product decision, however good the number looks.
- 3Is there a reference implementation, or just a paper? Research without reproducible code takes months longer to validate than research that ships with a working example.
How a paper actually reaches production
What actually made it through this cycle
Long-context retrieval quality kept improving in ways that held up in our own testing, not just the labs' — models genuinely got better at using information buried in the middle of a long context, not only at the start and end. That one is real, and it's already changed how we design RAG pipelines that used to over-invest in aggressive chunking to compensate.
Smaller, cheaper models closing the gap on well-scoped tasks is also real and reproducible — for classification, extraction and structured-output tasks specifically, not open-ended reasoning. We've swapped several production tool-call classifiers to smaller models this year with no measurable quality drop and a meaningful cost cut.
Several "agentic self-improvement" and "automatic prompt optimization" techniques that looked strong on paper produced inconsistent, hard-to-debug behavior in our internal prototypes — the kind of unpredictability a client product can't absorb, whatever the benchmark said.
How we actually decide to prototype something
A research claim earns a one-week internal prototype, scoped to a single narrow use case, before it earns a place in any client conversation. Most stop there — quietly, without drama — because the gap between benchmark and product reveals itself fast once real data and real latency constraints are involved.
~15
papers we scan in a typical month
2–3
that earn an internal one-week prototype
1 in 5
prototypes that make it into an actual client build
“A benchmark tells you a technique can work. It never tells you what it costs, or how it fails — that's what the prototype is for.”
Key Takeaways
- Filter research through three questions: does it hold outside its own benchmark, what's the real cost/latency trade-off, and is there reproducible code.
- Long-context retrieval quality and smaller-model performance on well-scoped tasks are the two trends that held up under our own testing this cycle.
- Agentic self-improvement and automatic prompt-optimization techniques looked strong on paper but produced unpredictable behavior in practice — not yet client-ready.
- Every promising claim earns a narrow, one-week internal prototype before it's allowed anywhere near a client conversation.
Where to start
If you're evaluating a new technique for your own product, don't start with the paper's benchmark. Start by asking what it would cost to run at your actual traffic — that number kills more bad ideas than any amount of reading.
