When There Is No API: Driving Chrome With the DevTools Protocol, and When Not To

Every scheduled agent we run has eventually hit the same wall: the data is on a page, and there is no API behind the page. A freelance marketplace renders filtered listings only after a JS filter panel settles. A public procurement portal builds its results table from an XHR that needs a session cookie minted by the landing page. A newsletter site hides its archive behind infinite scroll.

The reflex is to launch a headless browser. Sometimes that is right. More often it is the most expensive correct-looking decision on the table, and the bill arrives three weeks later at 4am when the cron job OOMs the runner.

What the protocol actually buys you

The Chrome DevTools Protocol is a WebSocket JSON-RPC interface into a running Chrome or Chromium. You connect, you enable domains — Page, Network, Runtime, DOM, Fetch — and you get events and commands for each. Playwright and Puppeteer are wrappers over this; you can also speak it directly, and for small agents that is often less code than the wrapper.

The part worth internalizing: you usually do not want the DOM. Enabling Network and listening for Network.responseReceived, then calling Network.getResponseBody with the request ID, hands you the exact JSON the site’s own frontend consumed. The browser is doing the work you actually needed — executing the auth handshake, running the JS that constructs the request, holding the cookies — and you are reading the clean payload instead of parsing rendered markup.

We have automated 11 no-API targets this way. Eight of them resolved to intercepting a single JSON XHR. Only three genuinely needed DOM reads, and those three are the ones that break. Across roughly nine months of nightly runs, the DOM-reading jobs broke six times from markup churn — a renamed utility class, a wrapper div, a lazily-hydrated section. The XHR-intercepting jobs broke twice, both times because a response field was renamed, and both times the Zod schema at the boundary failed loudly instead of silently writing nulls.

The cost side is not subtle. In our runner, a Chromium instance idles around 180 MB RSS and peaks between 400 and 500 MB on a heavy page. Browser cold start is about 1.1 seconds; the target pages take another 2–6 seconds to reach a usable state. The equivalent fetch against a real API returns in roughly 40 ms. The container image goes from about 90 MB to around 400 MB once the Chrome shared libraries are in it. You are paying two orders of magnitude in wall clock and a full order in memory for the privilege of running someone else’s JavaScript.

The four failure modes that only show up on a schedule

Browser automation that works on your laptop and dies on a cron is not a mystery. It fails in four specific ways, and each has a boring fix.

Leaked browsers. If your process dies between launching Chrome and closing it, the Chrome stays. Do this nightly for two weeks on a small box and you get an OOM kill that has nothing to do with the run that triggered it. The fix is a preflight step that kills orphaned browser processes belonging to your job before launching a new one, plus a hard per-run timeout that escalates to SIGKILL. Track the browser PID in a file you own; do not trust the library to clean up after a hard crash.

Nondeterministic waits. networkidle is the most attractive wrong answer available. It never fires on pages with analytics beacons, polling, or a websocket, so your job hangs until the timeout instead of failing fast. Wait on a predicate you actually care about — the specific network response, or a selector that only exists once the real content is there.

Silent partial success. The page loads its shell, the list stays empty, your extractor returns zero rows, and the pipeline records a successful run. This is the worst one because nothing alerts. Assert cardinality: if a page that has returned 40–60 items every night for a month returns 3, that run failed. Pick a floor and fail below it.

State drift. Sessions expire, login flows get redesigned, an interstitial appears. Keep auth refresh in a separate step from extraction and give it its own exit code, so “we could not log in” pages someone and “the markup changed” opens a ticket.

Our incident log for that nine-month stretch: 23 failed runs total — 9 memory or zombie-process related, 6 wait-condition timeouts, 5 auth expiry, 3 markup changes. The first two categories are 65% of the failures and neither has anything to do with the site you are reading. They are operational bugs in your own harness.

When it is the wrong call

Driving a browser is the wrong call more often than the tooling ecosystem suggests. Concretely:

That last row deserves being explicit. Once a site has deployed a bot-detection product, the remaining engineering is evasion, and evasion is an arms race you will lose on a schedule — your job breaks on their release cadence, not yours. It is also a clear signal about consent. Read it as one.

The honest heuristic we use now: browser automation is justified when the target is small (tens of pages, not thousands), the cadence is slow (daily, not per-minute), the access is permitted, and the value of the data clears the roughly 100x cost multiplier over a plain HTTP call. Three of our 11 targets have since been retired because they stopped clearing that bar.