We put thirteen models and a keyword rule on the two safety decisions a medical-practice chat makes before it answers. One model caught every message from a writer at risk, and flagged every safe message too. What the board can rank, what it cannot, and why half of it is sealed.
Three vector indexes tied at about 0.90 nDCG@10 on a public benchmark. The chat path built on them did not, and the whole gap was questions that reached the model with no context.
We swapped a 70B chat classifier for a typed-decision model and got 7x the speed at higher accuracy. The result worth writing down was the one where both models scored exactly the majority-class baseline.
Seven of eight major agent benchmarks were driven to near-perfect scores without solving a task — and every one of those runs would have passed a signature check. The unit of trust is not the number. It is the manifest, and a manifest has to be able to state its own weakness.
EvaluationAttestationBenchmarksRAGContent ProvenanceC2PAOpen StandardsEU AI Act
An e-commerce assistant linked products correctly but recommended the wrong ones. The fix was discovering that decoration, ranking, and recommendation are three different problems — and that prominence is only ever a tiebreaker on relevance.
RAGProduct RecommendationsE-commerceBM25ProminenceStorefrontLLM GenerationBuilding in Public
We swapped OpenParse for LiteParse in our RAG ingestion pipeline. The headline is 20–50× faster parsing. The useful part is the four ways we got it wrong first.
Open-source Headroom compresses RAG context with an ONNX model. We wired it into our pipeline and raced it against a 50-line in-process extractor. Neither won outright.