AI QA on Vercel preview deployments: the three problems nobody writes about
Every Vercel preview deployment is a production-like environment you're probably not testing. Here's the honest wiring — the ephemeral URL, Deployment Protection, and retargeting an agent at a URL that didn't exist 90 seconds ago.
AI QA on Vercel preview deployments: the three problems nobody writes about
You already pay for the hardest part of continuous QA and you're almost certainly throwing it away.
Every push to a branch on Vercel builds a complete, production-like copy of your app at its own URL. Real build, real serverless functions, real environment variables, real database if you wired one up. It exists for as long as you need it and costs you nothing extra. It is, functionally, a staging environment that spawns itself per pull request — the thing teams used to spend a quarter and a platform engineer building.
And then most teams open it, click around the one page they changed, and merge.
Running AI QA on Vercel preview deployments is the obvious fix, and the pitch writes itself: an agent walks your critical flows on every PR, before merge, on the exact code in that PR. What nobody writes about is the wiring. The three-line version — "point your test tool at the preview URL" — skips every part that actually bites. So this is the honest version, including the parts where you have to make a decision instead of copying a snippet.
There are exactly three problems. None of them are hard once you can see them; all three are silently fatal if you can't.
Problem 1: you don't know when the preview is ready, or where it is
A preview URL is generated at build time and it's different for every deployment. You can't hardcode it, and you can't test it before the build finishes. So the first thing any preview-time QA needs is an event that fires when the deployment is genuinely ready and carries the URL with it.
Vercel documents two ways to get that event, and their knowledge base article on running end-to-end tests after a preview deployment lists both: GitHub Actions repository_dispatch events, or a webhook for the deployment.succeeded event if you're on another CI provider.
The repository_dispatch path is the one to reach for if you're on GitHub. Vercel fires a vercel.deployment.success event at your repository, and the payload carries the two things you need:
Inside that workflow, github.event.client_payload.url is the deployment URL and github.event.client_payload.git.sha is the commit it was built from. Vercel's own example checks out that SHA and passes the URL to Playwright as BASE_URL.
Two things worth internalizing here, because they're the difference between a check that works and a check that lies to you:
The event fires on deployment success, not on push. This is correct and it's why polling the deployment yourself is a worse idea than it looks. If you trigger tests on push and then sleep-and-hope, you're racing the build. Sometimes you test the previous deployment, which is the worst possible failure mode: a green check on code you never ran.
The SHA in the payload is your source of truth. Check out client_payload.git.sha, not main, not the PR head at workflow-start time. If someone pushes twice in thirty seconds you'll get two events, and each one needs to be pinned to the commit it was actually built from.
If your reaction to "an event fires and then a check races the build" is that this sounds like a flakiness generator, you've read why E2E tests are flaky correctly. The fix is the same as it always is: remove the race, don't add a sleep.
Problem 2: Deployment Protection will lock your agent out, and the standard fix doesn't work for browsers
This is the one that eats an afternoon.
If your Vercel project has Deployment Protection turned on — Vercel Authentication, Password Protection, or Trusted IPs — your preview URL is not publicly reachable. Anything automated that hits it gets an auth challenge instead of your app. Vercel's KB says this plainly: if you have Deployment Protection enabled, "ensure you use Protection Bypass for Automation so your test environments can reach your deployments."
Vercel's Protection Bypass for Automation doc describes a secret you present as either an HTTP header or a query parameter named x-vercel-protection-bypass. Vercel sets one secret as the VERCEL_AUTOMATION_BYPASS_SECRET system environment variable in your deployments. The header is the recommended method, and their Playwright example does exactly that — sets extraHTTPHeaders in the config.
Here's the part that matters for an agent, and it's the reason this section exists: a browser agent doesn't set request headers. It navigates.
That's not a limitation to apologize for, it's what the thing is. A QA agent drives a real browser the way a person does — it goes to a URL, it looks, it clicks. There's no config file where you inject extraHTTPHeaders into a human. And Vercel's docs are already ahead of you on the general shape of this problem: the header approach has a known gap even for Playwright, because "headless browser tools send the bypass header on the initial page load, but subsequent in-browser navigations don't include custom HTTP headers."
Vercel's answer to that is the query parameter plus a cookie:
To bypass authorization on follow-up requests (e.g. for in-browser testing) you can set an additional header or query parameter named
x-vercel-set-bypass-cookiewith the valuetrue. This will set the authorization bypass as a cookie using a redirect with aSet-Cookieheader.
And on the query parameter method itself, the doc is explicit that it exists precisely "for tools that cannot set custom headers."
So for a browser agent the URL you hand it is:
First navigation carries the bypass in the query string, Vercel redirects and drops a cookie, and every click after that authenticates off the cookie. The agent never has to know any of this happened. That's the whole trick, and it's about four hours of confusion compressed into one line.
Four honest caveats before you paste that anywhere:
- The secret is in a URL. Vercel recommends the header method specifically because "headers keep the secret out of URL logs, browser history, and referrer headers." You're taking that trade deliberately because your tool is a browser. Keep the secret in a CI secret, never in a Scenario prompt, never in a committed file, and be aware it will appear in your app's own access logs.
- If you're loading the app in an iframe, Vercel's doc says to set
x-vercel-set-bypass-cookietosamesitenoneinstead oftrue, because theSet-Cookieheader defaults toSameSite=Lax. - Rotating the secret invalidates old deployments. Per Vercel: "When you build a deployment, Vercel sets the environment variable value, so regenerating or deleting the secret in the project settings will invalidate previous deployments. You will need to redeploy your app if you update the secret in order to use the new value." Rotate the secret and your open PRs' previews go dark until they rebuild. Vercel now supports multiple bypass secrets per project, which is the escape hatch — one per tool, rotate independently.
- The bypass is not a skeleton key. Vercel is clear that it won't override active DDoS mitigations, rate limits applied during detected attacks, or attack-triggered security challenges. If your preview is behind an active mitigation, the bypass won't save you, and honestly it shouldn't.
Problem 3: the agent's URL is stored, and the preview's URL is not
This is the one that's specific to how agent-based QA works, and where the honest answer costs you a design decision.
In Monito, a Test Scenario holds a prompt and a URL. That's a deliberate design: the Scenario is a durable, reusable thing that belongs to a Project, and it's the reason a Scenario survives a frontend rewrite that would have shredded a selector-based spec. It's also why pointing one at a hostname that was generated ninety seconds ago takes an extra step.
The Scenario URL is a stored field, and scenario run doesn't take a per-run URL override. So the preview workflow is: retarget, then run.
One wrinkle worth naming: VERCEL_AUTOMATION_BYPASS_SECRET is a Vercel system environment variable — it's available inside your deployment, not inside your GitHub Actions runner. Your workflow needs its own copy, so add the same value as a GitHub repository secret. Same name below, two different systems.
project run waits for every Scenario in the Project and, per the CI guide, "exits with code 1 when any scenario fails, so the workflow fails automatically. It also writes a compact GitHub job summary with scenario results and links to Monito evidence." That job summary is what turns this from a log you never read into a PR check with the failure on it.
Now the design decision, stated plainly because pretending it away is how you get a flaky check:
Scenario URLs are shared state. If two PRs deploy at the same time and both workflows retarget the same Scenario, they will fight, and one of them will test the other's preview. That's not a hypothetical — it's the default outcome on any repo with more than one active PR.
There are three honest ways out, and which one you want depends on how much you care:
- One Project per preview-testing lane. A dedicated Monito Project used only by the preview workflow, with a GitHub Actions concurrency group so exactly one preview check runs at a time. Simplest. Serializes your PR checks, which is fine at small team size and annoying at large.
- Pin to a stable alias instead of the per-deploy URL. If you test the preview of one long-lived integration branch rather than every PR, the URL is stable, the Scenario never needs retargeting, and you can use the documented setup path below with no custom workflow at all. This is the boring option and it's the right one more often than people admit.
- Create the Project per run.
project create --name "PR-123" --url "$TARGET" --tag development, discover or create Scenarios against it, run, and tear down. Fully isolated, no shared state, more moving parts and more credits.
Start with the concurrency group. Move off it when it actually hurts.
The setup path for the stable case
If you land on the stable-environment version — and again, most teams should — you don't write a workflow at all, and you don't put a Monito API key in GitHub. After two one-time browser sign-ins:
your coding agent can run the documented plan/apply/verify flow from the CI integration guide:
plan is read-only and tells you exactly what it will write and bind. apply generates .github/workflows/monito.yml and registers a project-scoped trust binding for the repository's immutable ID, owner ID, workflow ref, triggers, and environment. verify --remote-auth proves the OIDC exchange works without running Scenarios or spending credits. At runtime GitHub issues a short-lived OIDC token to the exact trusted workflow, the CLI keeps it in memory and never prints it, and jobs from fork pull requests are skipped.
If you're writing your own repository_dispatch workflow instead, run monito ci plan first anyway. It's read-only, and it will tell you the exact binding it wants for your trigger set rather than leaving you to guess whether your custom workflow can use OIDC or needs the compatibility API-key path.
"Why not just test staging?"
The reasonable objection, and worth answering properly rather than waving away, because for some teams staging genuinely is the right answer.
Staging is one environment with a queue in front of it. Five PRs merge, staging redeploys, the nightly suite goes red, and now you own a bisect: which of the five did it? You'll figure it out — you'll spend forty minutes and a Slack thread figuring it out, and the person who broke it has already context-switched to something else. That's the tax, and you pay it every time, silently, and nobody ever writes it down as a cost.
A preview check inverts that. One deployment, one commit, one diff, one author, and they're still looking at the PR. The bug is found by the person who caused it, while they still remember why they did it. That's not a marginal improvement in cycle time — it's the difference between a bug being a five-minute fix and a bug being a ticket.
There's a second, less obvious argument. A preview check runs before merge, which means a red check is a conversation about whether to merge. A staging check runs after merge, which means a red check is an incident, however small. Same bug, same detection, entirely different organizational weight. Teams that move QA to preview time report the check being less stressful, and that's not a soft benefit — an alarm that means "don't merge yet" gets read, and an alarm that means "prod-adjacent thing is broken" gets muted.
Where staging still wins: anything needing production-like data volume, anything needing your real CDN and cache config, anything with a long-running migration, and anything where the flow crosses services that don't have previews. Previews are a copy of your frontend and your functions. They are not a copy of your infrastructure. Run both — the preview check is a gate, the staging check is a safety net, and they're catching different things. If you want the fuller version of that argument, how much does QA cost works the economics rather than the workflow.
What to actually test on a preview
A preview check is not your whole suite. It's a gate, and gates should be cheap and mean something. Each Test Run is roughly 8–13 credits — about $0.08–$0.13 — so cost isn't the constraint; time to a red check is. Three or four Scenarios that cover the flows where a regression would embarrass you is the shape you want.
The right selection is the flows that break when someone touches shared UI: sign-up, login, the primary create-a-thing path, checkout if you take money. If you want the reasoning behind picking flows rather than pages, what an AI QA engineer actually replaces works through it properly.
The thing a preview check gives you that a staging check can't: the failure is attributable. One PR, one deployment, one commit SHA, one Test Run. When it goes red, nobody has to bisect — the Monito Session has the screenshot timeline, console output, network log, and the agent's step-by-step reasoning, attached to the deployment that came from exactly one diff. That's the entire value proposition of shifting QA to preview time, and it's worth more than the coverage.
And because the Scenarios are prompts rather than selectors, the preview check doesn't fail on the PR that renamed a component — which is the failure mode that gets scripted preview suites disabled within a month of being introduced. Playwright vs AI testing covers where each approach earns its keep; the short version is that a check nobody trusts gets ignored, and a check that breaks on refactors becomes a check nobody trusts.
The honest limits
Things this setup does not do, so you don't find out at 3am:
- Preview databases are not production data. Your preview probably points at a shared staging database or a branched one. Either way, an agent creating a user on every PR is an agent filling that database with users. Have a cleanup story or use disposable data.
- A failing preview check is only as trustworthy as your seed data. If two concurrent PR runs share a database and both create
test@example.com, you get a red check that isn't a bug. Unique per-run data, always. - The bypass secret is deployment-bound. Covered above, but it's the one that will confuse you six months from now when you rotate the secret and every open PR goes red simultaneously.
- This doesn't replace testing production. Previews aren't production infrastructure, don't have production data volume, and don't have your CDN config. Keep the production smoke check.
Wire it up
Here's the Scenario to start with — the one that catches the most on the fewest credits. Create it against your stable preview or staging URL first, confirm it passes, and only then wire the retargeting workflow around it:
Save it as a Test Scenario, add the retarget-then-run step to a repository_dispatch workflow, and every PR gets a signup check on its own deployment before anyone reviews the diff.
Your first run is free — point it at your latest preview URL and find out whether the branch you're about to merge can still sign a user up.