How Do We Run Agent Browser Control Reliably in Production?

The principle behind running browser agents in our operations is one line: "reliability comes from constraining autonomy, not maximizing it," because pure autonomy wobbles in production, so we freeze stable paths as deterministic scripts and reserve AI for the dynamic parts (hybrid). The measurements bear it out - one open agent lifted its success rate from 30% under full autonomy to 80% with a human confirming the plan. And three disciplines: (1) perceive pages via the accessibility tree (structure), not pixels, (2) since prompt injection cannot be prevented, contain it (defense in depth), and (3) put a human checkpoint on irreversible actions. Browser control is an area we have worked deeply - 150 facts in memory - yet 85% are stale, a sign of how fast the terrain shifts.

In one line: browser-agent reliability comes from narrowing autonomy, not widening it - freeze stable paths as scripts, perceive via the accessibility tree, and leave irreversible actions to human approval.

In plain terms: operating a browser agent is like handing a new hire a company account. Give repetitive work by the manual (scripts), delegate only what needs judgment, and require manager approval for payments or deletions. You trust it by narrowing scope, not granting unlimited power.

How do we control the browser?#

A hybrid architecture plus accessibility-tree perception. We freeze stable, high-frequency flows like login and checkout as Playwright scripts and call the LLM only when a page changes or is unfamiliar ("in the loop only when needed"). Perception uses the accessibility tree, not screenshots - a structure of role and name (e.g. button "Submit") that survives CSS changes, is fast, and is deterministic without a vision model. But refs are valid only within a snapshot, so re-snapshot after navigation and filter out off-screen elements.

How do we stop injection and handle auth?#

Both defense-in-depth and least privilege. Prompt injection cannot be fully prevented (white text or HTML comments hidden in a page hijack a "summarize this") - so we shrink the blast radius with layers: session isolation (single-use VM, network segregation), marking external content (spotlighting), tool policy, and user confirmation. For auth, we never expose credentials to the agent - just-in-time injection, encrypted stored session, MFA, least privilege, and instant token revocation on anomaly. One rule: sensitive tasks only on trusted sites, and never keep email or banking tabs in the same session.

Where do humans go?#

On irreversible actions and low-confidence situations. Agents do single-step, supervised work well, but multi-step autonomy needs human checkpoints - a confidence threshold splits auto from human, uncertainty pauses to show reasoning and request review, and ambiguity falls back to a safe deterministic script. Irreversible actions like payment, deploy, and delete are left to human approval (with an audit log). One caution: a checkpoint is only as good as its reviewer - a rubber-stamping reviewer is worse than none.

So what is our operating discipline?#

The key is constraining autonomy and putting policy and isolation around every action.

• Structure: ==stable = script, dynamic = AI== (hybrid), perceive via the accessibility tree, re-snapshot after navigation.
• Security: handle injection with containment (single-use sessions, source marking), credentials just-in-time and least-privilege, humans on high-risk.
• Cost: snapshots eat tokens - prefer disk-backed snapshots (27K) over streaming every step (114K for a 10-step task) to save context. Measure reliability on your own metrics.

Related reading: 오픈 음성 클로닝, 우리는 이렇게 운영한다 — 파이프라인 회고, 에이전트 브라우저 제어, 직접 써보고 느낀 점과 한계

Related reading: 에이전트 브라우저 제어, 직접 써본 실측·한계, 터미널 AI 에이전트, 직접 굴려본 실측·한계

Reference links

• Playwright (browser automation, accessibility tree)
• Playwright MCP (structured browser tools)
• browser-use (LLM browser agent)
• Stagehand (hybrid primitives on Playwright)
• OWASP LLM Top 10 (prompt injection)

Note: success-rate, token, and stale figures are public 2026 measurements and our own observation, and vary by site, tool, and version (not permanent numbers). Injection defenses keep evolving, so re-review sensitive tasks against your own threat model (these numbers are only a start). The browser-agent terrain moves fast, so this is reviewed quarterly.