Auditing a 660-Article Archive for Near-Duplicate Content: Our Three-Pass Dedup Process

An archive does not develop a duplicate content problem in one bad week. It gets there one reasonable decision at a time: a topic queue surfaces the same launch from Hacker News on Monday and from Bluesky on Thursday. A comparison post gets written as “A vs B” in March and “B vs A” in September. A yearly refresh ships as a new slug instead of an update to the old one. None of those choices is wrong in isolation. Stacked 660 times, they produce an archive where you genuinely cannot tell whether a given draft already exists.

We ran a full near-duplicate audit across our own archive. What follows is the process, the thresholds, and the part nobody writes about — deciding what to actually do with a cluster once you have found it.

Why near-duplicates cost you something

Google’s own documentation is clear that duplicate content is not a penalty in the ordinary case. What happens instead is consolidation: the crawler picks one URL as canonical and the others stop earning their own impressions. That is not a fine, but it is still a loss. You paid to write four articles and you are being served as one.

The sharper cost shows up in retrieval. Answer engines chunk your pages and rank chunks by similarity to a query. If six of your articles contain a near-identical paragraph explaining what an agentic CLI is, those six chunks compete with each other for the same slot. Redundancy inside your own corpus is self-inflicted dilution, and unlike a ranking drop it produces no obvious signal in Search Console.

The third cost is editorial. When you cannot answer “have we covered this?” in under a minute, your topic pipeline starts generating work you have already done.

The three-pass audit

We ran three passes, cheapest first, each catching a different failure mode. Running them in this order matters — the expensive pass only has to look at what the cheap passes could not resolve.

Pass 1 — normalized hashing. Strip frontmatter, strip MDX component tags, collapse whitespace, lowercase, then SHA-256 the remaining body. Exact matches after normalization mean a file was copied. This found nothing in our archive, which is the expected result. Run it anyway: it takes seconds and it rules out the embarrassing category before you start interpreting fuzzy scores.

Pass 2 — lexical overlap via shingles. Break each normalized body into overlapping 5-word shingles, hash them into a MinHash signature, and compare signatures by estimated Jaccard similarity. This catches copy-paste: a methodology section reused verbatim, an intro paragraph lifted from a sibling post, a pricing table pasted into three reviews.

One detail decides whether this pass is useful or noise: strip shared boilerplate first. Disclosure blocks, AI-assistance notes, newsletter copy, and standard closing sections are identical by design. On a 900-word post, boilerplate can be a third of the token count, which pushes every short article into apparent similarity with every other short article. We excluded any block that appears in more than five percent of the archive before computing signatures.

Pass 3 — semantic similarity via embeddings. Embed each article body, then compute pairwise cosine similarity. This is the pass that finds the real problem: two articles making the same argument, in different words, with different examples. Lexical overlap on those pairs can sit near zero.

At 660 articles the full pairwise matrix is roughly 217,000 comparisons, which is trivial in NumPy. The embedding call itself is the only meaningful cost, and at a couple of thousand tokens per article you are embedding well under two million tokens total — a rounding error against any provider’s embedding pricing.

Here is how the three passes compare on what they catch and what they cost:

Triage is the actual work

Finding clusters took an afternoon. Deciding what to do with them took considerably longer, because a high similarity score is a question, not a verdict. We sorted every cluster into one of four outcomes.

Merge and redirect. Two articles answer the same question for the same reader. Pick the winner by inbound links and existing impressions, not by which one you like better or which is newer. Fold any unique material from the loser into the winner, then 301 the loser’s URL. Deleting without a redirect throws away whatever authority the page had accumulated.

Differentiate. The overlap is real but the articles have legitimately different jobs — one is a hands-on review, the other a buying comparison that happens to restate the same setup context. Here you rewrite the overlapping section in one of them rather than merging, and add explicit cross-links so each page points at the other for the part it does not cover.

Keep both. The score is high because the corpus is narrow. Two independent reviews of two competing tools in the same category will always look similar to an embedding model. If a reader searching for one would be annoyed to land on the other, they are not duplicates. This bucket was larger than we expected, and it is the reason a purely automated dedup pass is a bad idea.

Delete. Reserved for pages with no traffic, no links, and nothing worth folding into the survivor. Still redirect to the closest relevant page.

One rule saved a lot of arguing: the article with the most inbound links wins by default, and overriding that default requires a stated reason. Without it, every cluster turns into a discussion about writing quality.

Stopping the archive from re-accumulating

A one-time audit buys you a clean archive and nothing else. The drift resumes the next time the topic queue runs.

The fix is to move the check upstream. We embed each draft before publish and compare it against the archive’s existing vectors. If the nearest neighbor scores above our calibrated threshold, the pipeline stops and prints the matching slug and its score. Most of the time the right response is to update the existing article instead of publishing a new one, which is usually the better SEO outcome anyway.

Two things make this practical. First, cache the archive’s embeddings and key them on a content hash, so a re-run only embeds what actually changed. Second, log the nearest neighbor and its score on every publish, even when it passes. That log is what lets you re-calibrate the threshold later using real data rather than re-guessing.

We re-run the full pairwise audit quarterly. The pre-publish check catches direct restatements; the full audit catches slower drift, like a category where six articles converge on the same framing over a year without any single pair tripping the gate.

The audit is worth running once even if you never automate it. Our most useful finding was not any single duplicate pair — it was discovering which categories the topic pipeline kept circling, which changed what we queued next.