agentic-browser-testing

| Data | Seeded fixture + reset/seed the database before every run | | Scope | Bounded step budget (maxSteps), e.g. 18 — exceeding it FAILS, never auto-retries | | Oracle | Independent checks against fresh application observations | | Evidence | Assert on the accessibility tree, never a screenshot diff |

Avoid: temperature: 0.7 or 1 for exploration, retry-until-pass loops, waitForTimeout sleeps, and screenshot-based assertions. Each one hides flakiness rather than removing it. Adapter configuration examples are in references/setup.md; these fields need an actual host adapter to enforce them.

Success Assertion: the Oracle (where agents fail silently)

This is the sharpest failure mode: the agent reports success while stuck on the login page, because "page loaded / no error / looks good" was accepted as success and the LLM was allowed to self-grade. Force an explicit oracle the harness checks — never the agent.

For the goal "sign in as an existing user and confirm the dashboard shows the right account name":

SUCCESS (all must hold — assert against the final browser_snapshot):
  - URL matches /dashboard
  - Snapshot contains the specific expected account name text, e.g. "Acme Corp — Jane R."
NEGATIVE / forbidden state (fail fast if any is true):
  - Still on a URL matching /login  → FAIL
  - Snapshot contains role="alert" with "invalid credentials"  → FAIL
VERDICT: a project-owned oracle executes these checks; the evidence gate emits passed:true only after fresh, complete checks. The LLM does not decide.

The positive checks (specific account name + /dashboard URL) prove where it landed; the negative check (must NOT be on the login page) is what kills the false pass. "No error," "didn't crash," "screenshot looks correct," and "trust the agent" are not success criteria.

Cost and Latency

Measure agent latency and cost against the scripted alternative. Model round trips can dominate a small browser flow. Control unnecessary work with explicit budgets:

• Step budget — keep maxSteps low and enforced; fewer round-trips, less drift.
• Model selection — use measured task accuracy, latency and cost to choose an available model. Pin the model version where the provider supports it.
• Prompt caching — cache the static system prompt, tool schemas, and goal; they repeat every run.
• Scope via a seeded entry point — one narrow goal per run, deep-linked past login instead of re-driving it each time.
• Snapshot size — scope snapshots to the relevant region where supported. Measure payload size; large DOMs can make text snapshots expensive too.

Backwards moves to reject: "use the biggest model for every step," "raise the step limit," "screenshot every step," and running with no budget at all. See references/setup.md.

Graduation and CI Gating

Promote a stabilized goal into a durable scripted test, and gate merges on the verdict. Both are detailed in references/graduation-and-ci.md; the essentials:

• Graduate with Playwright Test Agents (planner / generator / healer, shipped in Playwright v1.56.0). npx playwright init-agents --loop=claude. The planner writes a Markdown test plan to specs/<flow>.md; the generator turns it into tests/<flow>.spec.ts with role-based locators (getByRole, getByLabel, getByText) verified against the live DOM; the healer repairs broken locators. This is the promotion path — not "keep running it as an agent," not recorded clicks, not page.locator('xpath=...'), not data-testid-only.
• Gate CI with scripts/agent-result-gate.mjs from a full QA Skills checkout. It runs a reviewed oracle command and requires typed, nonempty evidence for the exact run, target and build. failed and blocked both exit nonzero. The host agent adapter must separately enforce step/time budgets; a model-produced {"passed": true} cannot pass this gate.
• Canvas with no accessibility tree: prefer instrumenting the canvas with ARIA; as a scoped last resort enable --caps=vision to unlock browser_mouse_click_xy for that flow only. browser_snapshot will not work on a raw canvas, but don't make coordinates the default and don't abandon agentic testing.

Migrating a brittle script to a goal (honest tradeoffs)

Converting an 80-line script that re-types login and walks 6 hardcoded steps into a single NL goal with an explicit success assertion is a real win for a churning flow — but state the downsides honestly:

• Non-determinism / false-pass risk — the run could pass falsely; that's why the hard oracle and the negative check are non-negotiable.
• Cost/latency — measure both; enforce a step budget, wall-clock limit and seeded entry point.
• Not every test should be agentic — keep stable paths scripted, and plan to graduate this one back to a scripted test once it stabilizes.

Reject the over-promise: it is not "strictly better with no downsides," do not "migrate everything," and never drop the assertions to make it pass.

Anti-Patterns

1. Reflexively writing a scripted Playwright test

"Browser test" pattern-matches to codegen, so the default is page.goto / page.locator / await expect(page...) / hunting data-testid. That misses the entire point. A goal-driven agent reads NL intent and explores via browser_snapshot — no pre-written selectors.

2. "Use the agent for everything"

Agent exploration has variable cost, latency and behavior. Stable critical paths (login) stay scripted and pinned; intent-driven wins on fast-changing UI. Never "always use the agent" or "agents replace all scripted tests."

3. Fixing flakiness with retries or higher temperature

"Just retry" and bumping temperature for "smarter" exploration both add variance. The real levers are temperature 0, a pinned model, seeded data, a bounded step budget, and an explicit verdict.

4. Assuming computer-use = screenshots + pixel coordinates

Playwright MCP is accessibility-tree-first. Defaulting to vision, screenshots, OCR, or mouse_click_xy is slower, costlier, and flakier. Pixels are a canvas-only last resort.

5. "No error = success" (the false pass)

Accepting "page loaded / didn't crash / looks good" and letting the LLM self-grade is exactly why the agent reports success while stuck on login. Require specific expected text, a URL, and a forbidden-state negative check, evaluated by the harness.

6. Bigger model / more steps to go faster

Increasing model size or step limits is not evidence of reliability. Compare actual results under the same task and fixture, then select budgets and a model from that evidence.

7. Running an agent forever instead of graduating

A goal green for two weeks on a now-stable flow should become a scripted tests/*.spec.ts via Playwright Test Agents. "Keep running it as an agent," recording clicks, and xpath locators are all wrong promotions.

8. Prose verdict in CI

Returning a paragraph for a human to read, or continue-on-error: true, lets a failed goal merge. The run must exit non-zero on failure with a machine-readable boolean.

Done When

• Goal prompt exists as natural-language intent (no page.locator / page.goto / data-testid in the goal) with a START seeded entry URL.
• An explicit success oracle is defined: specific expected text AND a URL check AND a forbidden-state negative check, asserted against browser_snapshot — not a screenshot.
• Run config pins a supported model version and sampling settings, a maxSteps budget, and a seed; no waitForTimeout, no retry-until-pass.
• Interaction is snapshot-first: browser_navigate / browser_snapshot / browser_click / browser_type / browser_wait_for; screenshots used only for evidence.
• The adapter emits a typed activity record; the independent gate executes the approved oracle and writes gate-result.json. Missing or blocked checks fail CI, as do failed ones.
• CI seeds/resets ephemeral state per run and enforces a step budget and a timeout cap.
• A graduation trigger is recorded (e.g. "green for 2 weeks → run init-agents, generate tests/<flow>.spec.ts with getByRole locators").
• If any target is canvas/WebGL, the vision fallback (--caps=vision + browser_mouse_click_xy) is scoped to that flow only, or the canvas is instrumented with ARIA.

Related Skills

• playwright-automation — Writing and maintaining deterministic scripted Playwright tests and Page Objects. Go there to author the durable test; this skill graduates an agent run into one.
• ai-system-testing — Testing your product's OWN LLM/AI features (prompt regression, model output quality). This skill tests any app using an agent; it does not test your AI feature.
• exploratory-testing — Human SBTM exploration and bug hunting. The agentic smoke goal is the automated cousin; use exploratory-testing for charter-driven manual sessions.
• test-reliability — Self-healing locators and quarantine for scripted flaky tests at runtime. Complements the determinism levers here once a test has graduated.
• qa-project-context — The universal dependency; supplies stack, environments, seed/reset tooling, and model access that every question above depends on.

Reference Files (in references/)

• setup.md — Playwright MCP registration (.mcp.json), the snapshot tool table, the natural-language goal prompt with success/negative assertions, the determinism harness config (pinned model, temperature 0, maxSteps, seed, prompt cache), and cost/latency levers.
• graduation-and-ci.md — Playwright Test Agents promotion pipeline (planner → specs/*.md, generator → tests/*.spec.ts with role-based locators, healer), the GitHub Actions gating flow using the executable independent evidence gate, and the canvas --caps=vision fallback.
• gate-contract.md — exact gate config, adapter activity record and oracle evidence schema, plus a bounded HTTP example. Requires a project-owned host adapter; no model runner is bundled.