Idempotency: Publishing Agents That Survive a Mid-Run Kill

A scheduled publishing agent is almost entirely I/O against services you do not own, on a clock you do not control. Ours fans out to four destinations per article: a search-engine index ping, a microblog, a federated social network, and a developer community site. That last one rate-limits hard enough that we pace requests 75 seconds apart and back off on 429s. A three-article run therefore spends over four minutes inside the fan-out, and most of those minutes are spent sleeping.

Four minutes is plenty of time to get killed. A CI job hits its wall-clock cap. A container gets evicted mid-sleep. Someone closes a laptop. The run dying is not the interesting part. The interesting part is what the next run does about it.

Retry is easy; knowing what already happened is not

The failure that costs you is not a crash — it is a partial fan-out that leaves no obvious trace. Three articles across four channels is twelve remote calls. The process dies on call seven. Article one is everywhere. Article two reached two of four channels. Article three exists nowhere but your git history.

Now pick a retry strategy. Re-run the whole job and article one gets posted a second time to every channel that has no server-side dedupe. Skip anything carrying a published flag on the article record and article two never receives its remaining two channels — not on the next run, not ever. Both strategies are wrong for the same reason: they track state at a granularity that does not match the work.

We shipped a worse version of this. For a long stretch, our publish command built and deployed the site but never invoked the syndication step at all. 57 articles went live and were never announced anywhere. Nothing threw. Exit code zero, every time. We found it by reading the script, not by reading logs, because there was nothing in the logs to read.

The rule that falls out of this: state belongs to the (item, channel) pair. Not to the run. Not to the item.

Design the ledger before you write the retry loop

Three properties do the real work, and none of them are the retry loop itself.

Write in two phases. Mark the pair in_flight before the remote call, resolve it to done or failed after. A kill that lands between the network write and the ledger write is not a hypothetical — it is the single most likely place to die, because that window contains the network. Without a two-phase record you cannot distinguish never sent from sent but unrecorded, and those two states demand opposite actions.

Store the identifier the remote system gave you. The post ID or URL that came back in the response is what lets you reconcile later without guessing. It also turns an ambiguous in_flight row into a question you can answer with a read.

Keep the ledger outside the run. Not process memory, not the job’s temp directory, not an in-memory queue that dies with the worker. A committed JSON file or a table. Ours lives in the repo, which means the diff shows exactly what shipped and when — the same reason we generate article metadata ahead of build time rather than during it.

With that in place, resume stops being a mode and becomes a filter:

// pending work is a query over the ledger, not a resume cursor
const pending = [];
for (const item of items) {
  for (const channel of CHANNELS) {
    const row = ledger.get(item.slug, channel.id);
    if (!row || row.state === 'failed') pending.push({ item, channel });
    else if (row.state === 'in_flight') pending.push({ item, channel, verify: true });
  }
}

A resumed run and a fresh run now take the same code path. There is no recovery branch to maintain and no --resume flag anyone has to remember at 2am. That matters more than it looks, because you cannot reliably test a recovery branch: the kill can land anywhere, and the cases you write tests for are the ones you already thought of.

When the API gives you no idempotency, buy it with a read

Publishing APIs rarely ship the Idempotency-Key header that payment APIs standardized years ago. In practice you land in one of three tiers.

The canonical URL is the natural dedupe key for anything content-shaped, and most channels let you search your own posts for it. Pay that read only when a row is stuck at in_flight — on a clean run it never fires, so the cost sits at zero in the common case and one request in the case that actually needs it.

One more classification is worth making explicit before you write any of this: decide, per channel, whether a repeat is harmless. Index pings are naturally idempotent, so ping freely and treat them as at-least-once. Social posts are public and permanent, so prefer at-most-once and accept a missed announcement over a duplicate — a missing post can be sent by hand tomorrow, a double post cannot be un-seen. Applying one policy uniformly across both kinds is how the same link ends up in a feed three times.

The refactor is mostly mechanical once the ledger schema is settled, and it is exactly the kind of repetitive, well-specified edit worth handing to a coding agent while you keep the schema decision for yourself.

None of this makes the agent more capable. It makes the agent’s failures cheap, which for anything running on a schedule is the property that determines whether you keep running it.