AI Translation and Localization QA: A Pipeline Product Teams Can Actually Staff in 2026

Localization used to be a quarterly batch job: freeze the strings, ship a spreadsheet to a vendor, get it back in three weeks, hope nothing changed. Most teams don’t work that way anymore. Strings get extracted on merge, machine-translated within minutes, and land in front of users the same day.

The translation half of that loop is the part that got cheap. The QA half didn’t. You still need to know whether the German label fits the button, whether the Japanese plural form survived the round trip, and whether your product name got helpfully translated into something that means nothing to anyone. Those are the failures that reach users, and none of them are caught by asking “is this a good translation?”

Where machine translation breaks in a product UI

UI strings are a hostile environment for translation models. They’re short, context-free, full of markup, and constrained by pixels. The failures cluster into a small number of shapes, and knowing the shapes is most of the work:

Placeholder and markup damage. {count} becomes {anzahl}, %s gets dropped, an <b> tag closes in the wrong place. This is the single most common class we see in raw MT output, and it’s also the easiest to catch mechanically — which is exactly why it should never reach a human reviewer.

Plural and gender rules. English has two plural forms. Russian, Arabic, and Polish have more, and a model handed an English source with one/other branches will frequently emit a target with the same two branches. It reads fine. It’s wrong for most numbers.

Concatenation. "Delete" + " " + itemType works in English and falls apart in any language with different word order or grammatical case. The translator sees two unrelated fragments and has no way to fix it. This is a code bug that surfaces as a translation bug.

Length. German and Finnish routinely run longer than English; CJK runs shorter but needs different line-break handling. A string that’s correct and 40% too wide is still a defect.

Terminology drift. Your app calls it a “workspace.” Across 400 strings the model renders it three different ways in the same locale. Each one is defensible in isolation. Together they make the product feel machine-made.

Register. Formal vs. informal address (du/Sie, tu/vous) is a product decision, not a linguistic one, and models default inconsistently unless told.

A three-tier pipeline

The useful mental model is that these failure classes have wildly different costs to detect. Sort the checks by cost and run them in that order.

Tier 1 — deterministic checks in CI. No model, no API call, no judgment. Parse both sides and compare: placeholder set parity, ICU message syntax validity, HTML tag balance, do-not-translate list (product names, null, brand terms), glossary term presence, character-length budget per key, encoding and directionality markers. These run in under a second on a full string set and they should block the merge, not file a ticket. Anything a regex or a parser can decide belongs here, permanently.

Tier 2 — an LLM review pass. This is where the judgment calls go: accuracy against the source, terminology consistency across the locale, register, locale conventions (dates, currency, number separators), and awkward-but-grammatical output. Give the reviewer more context than the translator got — the source string, the key path, the target, the character limit, the surrounding strings on the same screen, and a screenshot if you have one. Ask for structured output: one row per issue with category, severity, and a suggested replacement. Free-form prose review is unactionable at scale.

Tier 3 — sampled human review. You are not going to human-review every string, and you don’t need to. Sample by risk: 100% of legal, billing, and destructive-action copy; a fixed sample of everything else, weighted toward locales with the highest traffic and the highest tier-2 defect rates. The reviewer’s job is not to re-translate — it’s to confirm or overturn tier-2 verdicts, which is what keeps your automated scores honest over time.

The thing that makes this pipeline staffable is that tiers 1 and 2 do the volume, and the human budget goes to a sample small enough that one contract linguist per priority locale can keep up with continuous shipping.

Scoring it so the number survives an argument

“The translations look good” is not a release gate. Borrow the structure of the industry error taxonomies rather than inventing one: classify every confirmed issue by category (accuracy, terminology, locale convention, style/register, markup/format) and by severity (critical, major, minor).

Define severity by user impact, not by linguistic offense. Critical means the string misleads the user, breaks the layout, or corrupts data — a mistranslated “Delete permanently,” a broken placeholder that renders {count} literally. Major means the meaning is intact but the string is visibly wrong. Minor is style.

From there, the reportable metric is a defect rate — confirmed issues per 100 strings, split by locale and by surface (onboarding, settings, billing, error states). Two properties make it useful: it’s comparable across releases, and it tells you where to spend. A locale with a rising terminology-category rate needs a glossary update, not more review hours. A surface with critical-severity spikes needs a code fix, usually concatenation.

Set the gate as a policy, not a vibe: zero unresolved criticals in any locale, and a major-severity rate that doesn’t regress from the previous release. Everything else ships and gets fixed in the next cycle.

Keep a golden set: 100–200 strings per priority locale that a human has confirmed, covering each failure class at least once. When you swap translation models or change a prompt, run the golden set first. It converts “the new model feels better” into a number you can compare, and it’s the cheapest regression test in the whole pipeline.

The teams that get this right treat localization QA as a build step with a pass/fail condition, not a review meeting. The pipeline is boring on purpose: parsers catch what parsers can catch, a model catches what needs reading, and a human confirms a sample so the first two stay honest.