A Jaccard Gate at 0.50 Before the Model Runs

The gate that decides whether this site spends a generation is 93 lines of TypeScript with no dependencies. It refuses a topic outright when Jaccard similarity against any existing title reaches 0.50, prints a warning at 0.35, and before scoring anything it drops every token matching /^20\d\d$/. On the live corpus that year filter touches 78 of 274 published titles — 28% of everything on the site carries a four-digit year.

That last number is the whole reason the filter exists, and the reason it cuts in two directions at once.

The gate runs twice, and both times before something expensive

The generator calls assertNotDuplicate at two points. The first is the cheapest check in the pipeline: it runs against the topic string pulled from the candidate queue, before a prompt is even assembled. The second runs against the title the model actually produced, because a model handed a topic about connection pooling will cheerfully return an article about ORM query builders — one we already have.

The cost asymmetry is the entire argument. Gate one is a readdirSync over 274 files, a frontmatter slice, and a set intersection. The thing it guards is invokeClaude(prompt, 360_000) — a call configured with a six-minute timeout, followed by JSON extraction, zod validation, and an MDX compile pass. You do not need the gate to be clever. You need it to be free, and to run first.

This was added after a specific failure. In one day the generator produced five variants of the same tool review and three of the same benchmark, and Search Console came back with 60 pages classified as “Duplicate without user-selected canonical.” Nothing in the pipeline had ever compared a proposed topic against what already existed. The queue fed it topics; it wrote them.

Why the year token is stripped, and why it cuts both ways

Take two real titles from the corpus:

• “AI Code Review Tools Compared: CodeRabbit, Greptile, and Diamond in 2026”
• “AI Meeting Notetakers Compared: Granola, Fathom, and Otter in 2026”

After lowercasing, stripping punctuation, dropping tokens of two characters or fewer, and removing the 25-word stop list, each reduces to six content tokens. They share exactly one: compared. Union of 11, intersection of 1, so Jaccard is 0.09.

Now leave the year in. Each set grows to seven tokens, the intersection becomes compared and 2026, and the score is 2/12 — 0.17. The same unrelated pair, scored nearly twice as high, because both titles mention a year.

Title token sets are small. Six content words is typical here, so a single spurious shared token moves the score by roughly 8-9 points. With 28% of the corpus carrying a year, a proposed title that also carries one gets that free intersection against a large slice of everything you have already published. Enough of those stack up near 0.35 and the gate starts warning on articles that have nothing to do with each other — and a warning nobody trusts is a warning nobody reads.

The second direction is the one that surprised us, and it is the more useful half. Strip the year and “The Best Async Standup Tools in 2025” and “The Best Async Standup Tools in 2026” become identical token sets. Score 1.00. Hard abort. That is correct behaviour, not a bug to work around: a year-only difference is not a new article, it is an update to an existing one. The right move is editing the published post and adding a changelog entry, not shipping a second URL that competes with the first for the same query.

Both behaviours come from the same one-line filter. You cannot take one without the other, and you should not want to.

The stop list reinforces this. It holds best, review, guide, vs, how, and why — precisely the scaffolding a templated listicle title is built from. Strip that plus the year and two listicles get compared on their subject nouns alone, which is the only part that determines whether they are the same article.

0.50 and 0.35 are guesses that survived, and here is what we did not test

Both thresholds were picked to fire on the failure we had actually observed, not derived from a labelled set. 0.50 blocks; 0.35 warns; the score used is the higher of the title comparison and the slug comparison, since a model sometimes keeps the topic in the slug after rewriting the title away from it.

What we cannot tell you: the block rate, or the false-positive rate. The generator’s catch handler increments a single failed counter, and a duplicate abort and an MDX compile failure both land there identically. Nothing distinguishes them in the tally. We also did not run a full pairwise sweep across all 274 live titles for this article — the counts here are grep-verified and the two scores above are hand-computed from the tokeniser’s actual rules. If you build this, add a distinct counter for gate rejections before you tune the numbers, or you will be tuning blind.

The alternative worth naming is cosine similarity over embeddings of title plus description. The condition that flips the decision is the shape of your duplicates. Jaccard sees shared tokens and nothing else, so it scores “Postgres Connection Pooling With PgBouncer” against “Avoiding Connection Exhaustion in Supabase” at close to zero — no overlapping content nouns — even though the two answer the same question for the same reader. If that paraphrase case is what keeps slipping through, token overlap is structurally blind to it and you need vectors, plus the storage and refresh job that come with them.

If what keeps slipping through is a generator emitting five near-identical titles in one batch, Jaccard already catches it, runs offline, needs no API call, and has no index to keep in sync. That is the failure this site had. It stayed crude on purpose.

One boundary to keep in view either way: this gate reads titles and slugs. Body-level duplication — two articles with different titles making the same three arguments — is invisible to it, and no threshold you pick will change that.