Agent Browser Control, Previewed: What It Is and Why

An agent browser-control system is AI that, like a person, looks at a web page and clicks and types to finish a task. Our gateway calls the browser-control agents of Grok, Gemini, ChatGPT, and Genspark through a single call, returning the answer text together with URLs for any artifacts (images and files). We actually called gemini:fast and it returned a one-sentence definition instantly, with the response structured into text, image URLs, a conversation id, and a finish reason.

In plain terms: a browser-control agent is an intern holding a mouse and keyboard. Even on sites with no API, it looks at the screen and clicks and types to get the job done.

Start with the terms. An agent is an AI that, given a goal, takes several steps on its own to finish it. Add browser control and that agent opens a real browser like Chrome, looks at the screen, and moves the mouse to work. A gateway is a relay layer that wraps many providers' agents behind one shared interface, so you swap models with a one-line change.

What does it control, and how does it work?#

The key is being multimodal, reading pixels and the DOM together. Older automation bots leaned on per-site APIs or selectors, but modern browser agents perceive the screenshot (pixels) and element layout (DOM) at once, so they click and type on any page. Research agrees: vision-grounded agents that see the screen far outperform text/DOM-only models. That is what lets them automate human-in-the-loop flows like login, form filling, search, and booking.

The DOM is a tree-shaped data structure describing where every button, input, and link sits on a page; the pixels are how that page is actually painted on screen. People use both: we see the page (pixels) to decide where to click, and infer structure (DOM) when the layout looks off. Feeding an agent both, as below, cuts misjudgments.

By public benchmarks, performance measures out like this.

Plotting the WebVoyager scores as bars makes the top-to-bottom spread obvious.

Why a gateway instead of one model?#

Because each model has different strengths and locking into one is risky. Our gateway selects, through the same interface, grok:fast/expert/heavy, gemini:fast/pro, gpt-5, and genspark (a deep multi-step research super-agent). Step up to the expert/heavy variants for deep reasoning, or down to fast for quick answers. You continue a session by conversation id for multi-turn work, and you receive any images or files the agent produced via image_urls. Swapping providers is a one-line change, so there is no vendor lock-in.

In one picture, your code talks only to the gateway, and the gateway routes to the providers behind it.

Can you trust the benchmark numbers as-is?#

Be careful. WebVoyager scores in the 90s have become common, but a 2025 paper warns that as trajectories get longer, agreement between automated evaluation and human judgment drops sharply (too many screenshots, judging only the final frame). Some results also modified the original task set (for example, removing 55 tasks), so the numbers are not always apples-to-apples. Treat success rates as direction only and verify on real sites yourself.

How can you check it yourself?#

Measure on your own task.

• Send the same task (for example, "from this product page, get the price and review count") to the fast and expert variants and compare success/failure and time.
• If you need artifacts, confirm image_urls gets populated; for multi-turn work, chain the conversation id.
• Treat benchmark scores as a starting point, then measure 5-10 scenarios in your own domain to set a realistic confidence range.

Reference links

• Model Context Protocol (MCP spec)
• On evaluating web agents (paper)
• Browser Use (open-source SOTA)
• WebArena (benchmark)
• WebVoyager (paper)

Note: benchmark figures are public 2024-2025 measurements and vary with evaluation methodology and task set (compare with care). Gateway providers and variants change often, so verify exact behavior and performance by calling it yourself with the method above. Reviewed quarterly.