The Engram filter: 7 signals, one score, no false positives

Pure vector search is a great default until it isn't. The failure mode is predictable: semantically similar but topically wrong. how do I cancel my subscription retrieves the refund policy. what's the password reset flow retrieves the account deletion guide. Both pairs have high cosine similarity. Neither is what the user asked.

We hit this in production constantly. The fix wasn't a better embedding model — it was a re-ranking layer that combines semantic similarity with six other signals and gates on diversity.

We call it the Engram filter. It runs on every CAR-1 query and most CAR-1.5 queries.

The seven signals

Each candidate chunk gets a score from each signal, normalised to [0, 1]. The composite score is a weighted sum. The weights below are tuned per-tenant but the defaults work well across most workloads:

• ›Semantic similarity (35%). Cosine distance against the query embedding. The baseline signal.
• ›Lexical recall (30%). Fraction of query terms that appear in the chunk. Catches the cases where two passages are semantically close but the chunk doesn't actually contain the answer.
• ›IDF recall (18%). Same as lexical recall but weighted by inverse document frequency. Rare terms count more.
• ›Bigram overlap (12%). Overlapping 2-word phrases between query and chunk. Catches phrase-level matches that pure unigram recall misses.
• ›Lexical precision (14%). Inverse of how much of the chunk is not relevant. Penalises long chunks that happen to contain query terms by accident.
• ›Exact match (6%). Did the literal query string appear in the chunk? Small weight but a strong tiebreaker.
• ›Rank prior (2%). Where did the vector search rank this chunk? Slight pull toward the original ranking when other signals tie.

The zero-overlap penalty

The single most important rule in the filter isn't one of the seven signals. It's a hard penalty: if a chunk has zero query-term overlap, we multiply its score by 0.25.

This sounds aggressive. It is. But the pattern we kept seeing was vector search returning chunks that shared topic vocabulary with the query but didn't actually contain any of the specific terms the user typed. The user asks for "STAN-1-Mini latency" and gets back a chunk about "the inference performance of small models." Semantically similar, lexically empty.

The penalty doesn't filter them out entirely — sometimes those passages really are the best answer. But it pushes them down the ranking enough that better-grounded chunks rise to the top first.

Diversity gating

After scoring, before returning, we run a diversity pass. Each retained chunk has to be at least 22% novel against every chunk already in the result set, measured by Jaccard distance on a 5-gram representation. Near-duplicates get gated out.

This sounds like a small detail. It's not. Without diversity gating, the top-5 results for a popular question are often five paraphrases of the same source paragraph. The model sees redundant context, the user sees a wall of repetition.

Numbers

Across our internal eval set, Engram lifts retrieval F1 from 0.71 (vector search alone) to 0.91. The relative gain is bigger on adversarial queries — where vector search alone scores 0.52, Engram scores 0.84.

Adversarial here means: queries we've manually constructed to be semantically close to the wrong answer. Production queries are usually less hostile, but the bias matters: when something goes wrong, you want it to fail in a way users can debug, not in a way that confidently retrieves nonsense.

When you don't want it

Engram is overkill for some workloads. If your tenant has clean, well-structured documents and queries that closely match the document language, CAR-0 (pure vector search) is faster and almost as accurate. The Engram filter adds ~20ms of latency.

We default to CAR-1 (with Engram) for new tenants because that's where the failure-mode pain shows up. Customers can opt down to CAR-0 if their data shape allows it.