Canonical URLs vs llms.txt: What We Expose to Search Engines and What We Expose to AI Crawlers

Every article we publish ends up in at least six places: the canonical HTML page, a dev.to cross-post, a Bluesky card, a Mastodon toot, and three machine-readable files that no human is meant to open. All six carry the same sentences. Handled carelessly, that is a duplicate-content problem wearing a growth-hack costume.

The fix is not publishing less. It is deciding, per destination, which crawler you are talking to and what you want back. Search engines and AI crawlers want different things from the same bytes, so we hand them different artifacts and mark each one accordingly.

Search engines want one URL. AI crawlers want the text.

A canonical tag is a deduplication instruction. You are telling a search engine: several URLs serve this content, consolidate the signals onto this one. Google treats it as a hint, not a command, but it is the strongest hint you get, and it decides which URL accumulates ranking signals.

AI crawlers are running a different job. GPTBot, ClaudeBot, PerplexityBot and friends are fetching pages to extract text — for training, for retrieval, or for a live answer with a citation attached. They are not consolidating a link graph. What they benefit from is text that is cheap to parse and a stable URL to point back at.

So the two contracts are:

• Search engines: exactly one indexable URL per article, and every internal link agreeing on it.
• AI crawlers: the same content in a form that does not require DOM parsing, explicitly excluded from the index so it never competes with the article page.

On pickuma, article URLs are audience-scoped: /for-dev/<slug>/, /for-pm/<slug>/, /for-junior/<slug>/, /for-investor/<slug>/. The old flat /posts/<slug>/ paths 301-redirect to the audience path derived from the article’s audience frontmatter field. Internal links never hardcode either shape — they all go through a postUrl() helper, and a verification script runs as part of the build and fails it if any post is missing its redirect.

That last part matters more than the URL scheme itself. A canonical policy that lives in a style guide degrades the first time someone pastes a raw path into an MDX file. A canonical policy that fails the build does not.

The canonical rules we don’t break

Two rules, both learned the boring way.

Rule one: the cross-post canonical points home, and it stays clean. Our dev.to cross-posts set canonical_url to the pickuma article URL with no query string. The tracking link is a separate thing — a footer link in the body carrying utm_source=devto&utm_medium=crosspost&utm_campaign=blog. Those two jobs never share a URL.

The temptation to UTM the canonical is real, because you want to know how much traffic the syndicated copy sends back. Resist it. https://pickuma.com/for-dev/x/?utm_source=devto is, to a search engine, a different URL string from https://pickuma.com/for-dev/x/. You have declared the canonical to be a URL that your own site then canonicalizes somewhere else. At best the signal is wasted; at worst you have created the exact duplicate you were trying to prevent.

Rule two: syndication happens in the same run as publication. Our post-publish step fans out to IndexNow, Bluesky, dev.to, and Mastodon immediately after the build ships. The ordering is not cosmetic. If a syndicated copy is indexed days before the original, the canonical tag is arguing against an already-settled decision instead of informing a fresh one.

Why all three AI-facing files are noindex

A build step generates three files before anything else runs:

• llms.txt — an llmstxt.org-style index, category-grouped links.
• llms-full.txt — the whole corpus as plain text, with each article’s key takeaways inline.
• articles.json — a strict JSON index: URL, title, description, key takeaways, verdict, category, audience, type, tools, dates.

All three carry X-Robots-Tag: noindex in our headers config. The reasoning is short: these files are, by construction, the same sentences as the HTML pages. To a crawler that wants text without paying DOM-parsing costs, that redundancy is the entire point. To a search index, it is three more URLs competing with the article you actually want ranked.

noindex gives you both. The file is fetched, read, and used. It just never enters the index. If your CMS or host does not let you set response headers per path, this is a real constraint worth checking before you commit to the pattern — a <meta name="robots"> tag is not available inside a .txt or .json file, so the header is the only lever.

The key takeaways block on each article is the piece that ties this together. It is generated offline by a script that calls an LLM once per article, keyed on a hash of the title, description, and stripped body, and committed to a JSON file in the repo. Unchanged articles are skipped, so re-runs are cheap. Nothing is generated at build time — builds stay deterministic and offline, and every model-written sentence that ships is visible in a diff before it goes out.

Those same takeaways feed five surfaces: the rendered block on the page, BlogPosting.abstract in JSON-LD, a standalone ItemList node, the speakable selector, and both llms-full.txt and articles.json. One source of truth, five consumers, zero hand-editing.

What we can’t measure

Honest limits. Our working assumption is that answer engines pull disproportionately from the top of a page, which is why the compact answer sits above the body rather than as a closing summary. We cannot verify that assumption with our own data — nobody publishes per-paragraph citation attribution, and referral data from AI assistants is sparse and inconsistently labeled.

What we can verify is narrower and still useful: every article has exactly one indexable URL, every internal link resolves to it, every syndicated copy declares it, and the extractable version of the content exists at a stable path in three formats without polluting the index. Whether that earns more citations is a bet. Whether it prevents self-inflicted duplication is not — that part is just plumbing, and it either passes the build or it doesn’t.

If you are wiring this up on your own site, the order that matters is: pick the canonical URL shape first, enforce it with a link helper and a build check, then add the machine-readable layer on top. Adding llms.txt to a site whose canonical story is already ambiguous just gives the ambiguity another place to live.