GitHub Actions AI testing: make a red check block the merge, not just annoy you
Most GitHub Actions AI testing setups produce a green check nobody trusts. Here's the honest wiring — an agent that runs your real flows on every PR, an OIDC workflow with no stored key, a required status check that actually gates the merge, and the fork-PR footgun to avoid.
GitHub Actions AI testing: make a red check block the merge, not just annoy you
There are two things people mean by "GitHub Actions AI testing," and they are not the same product.
One is an AI that reads your diff and leaves review comments — a summarizer that guesses whether the code looks right. The other is an agent that opens a browser against the code in the pull request and finds out whether the app still works. The first is a nicer changelog. The second is a gate. This post is about the second one, and specifically about the part everyone skips: turning "an agent ran and it passed" into "this pull request cannot merge unless the agent passed." A green check that doesn't block a merge is decoration. A check that blocks a merge is QA.
That distinction is the whole game, because the failure mode of CI testing isn't "we didn't run the tests." It's "we ran them, the check went red, and someone merged anyway because the check is red half the time for reasons nobody trusts." A test suite that cries wolf gets branch-protection-exempted within a month. So the goal here isn't just to run an agent in Actions — it's to run one whose red actually means "don't merge," and to wire GitHub so that red is enforced rather than advisory.
We've written the Vercel preview-deployment version of this separately, because ephemeral preview URLs bring their own three specific problems. This post is the generic GitHub Actions path: a stable environment, the standard pull_request and push triggers, and the branch-protection settings that make the check load-bearing. If you deploy previews on Vercel, read both; the preview post handles the moving-URL and Deployment-Protection wrinkles this one deliberately leaves out.
What the workflow actually is
Strip away the marketing and a GitHub Actions QA check is four moving parts: a trigger (when does it run), a runner (where does it run), a credential (how does it prove who it is), and an exit code (how does GitHub learn pass or fail). Get those four right and the rest is prompt-writing.
Here's the shape Monito generates, straight from the CI integration guide. Read it once, because every design decision in this post is visible in it:
Two triggers. It runs on every pull request, and again on push to main after merge — the pre-merge gate and the post-merge confirmation. runs-on: ubuntu-latest is a normal GitHub-hosted runner; the agent's browser runs on Monito's infrastructure, not in the runner, so the runner is only orchestrating. The permissions block requests id-token: write, which is what lets the job mint an OIDC token — more on that below. And the one real step is project run --wait --github-summary.
--wait is the load-bearing flag. It means the workflow blocks until every Test Scenario in the Project has finished, and — per the CI guide — project run "exits with code 1 when any scenario fails, so the workflow fails automatically." That non-zero exit is the entire pass/fail contract with GitHub. GitHub doesn't know what a Test Run is. It knows that a step exited non-zero, so the job failed, so the check is red. Everything downstream — the PR status, the required-check enforcement, the merge block — hangs off that one exit code.
The step nobody documents: make it required
Here is where most "we added AI testing to CI" stories quietly end, one step too early. You add the workflow, you open a PR, a check appears, it goes green, everyone nods. And then the following week someone opens a PR where the check goes red, and they merge anyway, because nothing stopped them. A GitHub check is advisory by default. It reports; it does not block.
The setting that changes this lives in branch protection, not in your workflow file. In GitHub's own words, once you enable required status checks, "all required status checks must pass before collaborators can merge changes into the protected branch." You go to the branch's protection rule (or a repository ruleset), turn on Require status checks to pass before merging, and search for the Monito check by name to mark it required. From that point the merge button is disabled while the check is red. That — not the workflow — is the moment your AI test becomes a gate instead of a notification.
There's a subtlety worth knowing because it prevents a real spoofing class. GitHub notes that "any person or integration with write permissions to a repository can set the state of any status check," but when you mark a check required you can pin it to the specific app that's expected to report it: "If the status is set by any other person or integration, merging won't be allowed." Pin the check to the source you trust so a green status can't be forged by something else with write access. Small setting, real teeth.
One more decision the docs make you make on purpose. Required checks come in "strict" and "loose" flavors — strict requires the PR branch to be up to date with the base branch before merging, which means every merge to main re-runs the check on everything still open. Strict is safer and slower; loose is faster and occasionally lets through a bug that only appears when two independently-green PRs meet on main. At small team size, loose is fine and strict is annoying. Pick deliberately; don't inherit the default and wonder later why your queue is slow.
The fork footgun, and why the generated workflow has that if:
Look back at the workflow. This line is not boilerplate:
It says: run on pushes always, but on pull requests only when the PR comes from a branch in this same repository — not from a fork. That's there to protect you, and understanding why is the difference between a secure CI setup and a headline.
GitHub deliberately withholds your secrets from workflows triggered by pull requests from forks. Per GitHub's secure-use reference, with the exception of the automatically-scoped GITHUB_TOKEN, secrets are not passed to the runner when a workflow is triggered from a forked repository. This is correct and you want it: a fork PR is untrusted code, and if it could read your secrets, anyone on the internet could open a PR that exfiltrates them. So on a fork PR, a workflow that needs a credential to run tests simply can't get one — and rather than fail confusingly, Monito's generated workflow skips cleanly.
The dangerous "fix" people reach for is the pull_request_target trigger, which does run with secrets available — because it runs in the context of the base repository, not the PR. GitHub's own guidance is blunt about the trap: pull_request_target used with a checkout of the untrusted PR "can expose the repository to security compromises," and if you don't specifically need the privileged context, you should use pull_request instead. The number of credential leaks that trace back to someone swapping pull_request for pull_request_target to "make secrets work on fork PRs" is not small, and automated campaigns actively scan public repositories for exactly this misconfiguration. Do not be the repository they find. If you accept external contributions, gate the agent run behind a label or a maintainer re-trigger on trusted code — never hand fork code your credentials.
For the common case — a private repo, or a team pushing branches rather than forks — the if: guard is invisible and everything just works. It only ever matters the day an outside contributor opens their first PR, and on that day it does exactly the right thing.
No long-lived key in GitHub
The other reason that workflow is worth reading closely: there's no API key in it. No MONITO_TOKEN secret, no copy-paste credential, nothing to rotate or leak. That's the id-token: write permission and the MONITO_GITHUB_OIDC: "1" env var working together.
The setup is a three-command flow you run once, described in the CI guide: monito ci plan inspects your project and repo and prints exactly what it will do without changing anything; monito ci apply writes .github/workflows/monito.yml and registers a trust binding scoped to your repository's immutable ID, owner ID, workflow ref, triggers, and environment; monito ci verify --remote-auth dispatches an auth-only job that proves the OIDC handshake works "without running scenarios or spending credits." At runtime, GitHub issues a short-lived OIDC token to that exact trusted workflow, the CLI holds it in memory and — per the docs — "never prints or persists it," and the Monito authorization it buys "permits only CI status and run operations for the bound project." A leaked build log gives an attacker nothing, because there's nothing durable in the log to leak.
The long-lived-key path still exists, because some runners can't do OIDC, but it's explicitly the compatibility option, not the default: monito ci plan --auth api-key stores a MONITO_TOKEN GitHub secret, and the docs say plainly to "prefer OIDC for new GitHub Actions integrations." If you're wiring this up fresh in 2026, use OIDC and never think about a testing credential in your CI again. The whole category of "our test tool's API key ended up in a public log" stops being a thing you can do.
What to actually run on a pull request
A PR check is not your whole regression suite. It's a gate, and gates have to be fast and mean something, or people learn to wait them out. The constraint isn't cost — each Test Run is roughly 8–13 credits, about $0.08–$0.13 — it's time to a red check. Nobody blocks a five-minute merge on a twenty-minute suite; they merge and mean to check later.
So pick the three or four flows where a regression would actually embarrass you, and run those. The right selection is almost always the flows that break when someone touches shared UI or shared state: sign-up, log-in, the primary create-a-thing path, and checkout if you take money. Those are the flows where a renamed component or a broken API contract shows up as a real user-facing failure, and they're exactly the ones a scripted suite abandons after the third selector-rename false alarm. If you want the reasoning behind picking flows rather than pages or coverage percentages, what an AI QA engineer actually replaces works through it in detail.
The thing that makes this worth doing on a PR rather than nightly is attribution. A nightly suite that goes red is a bisect: five PRs merged, something broke, now find which one. A PR check that goes red is a sentence: this diff, this author, still on the page, still remembers why they wrote it. The bug gets found by the person who caused it while it's a five-minute fix instead of by whoever's on call three days later when it's a ticket. That's the same argument we make about why moving QA earlier beats catching it later, and it's most of the value — more than the coverage itself.
Reading a failure inside the PR
A check that only says "red" sends you off to a separate dashboard, and a check that makes you leave the PR to understand it is a check people stop reading. The --github-summary flag is the fix, and it leans on a native GitHub feature most workflows never use: job summaries. Per GitHub's workflow-commands reference, when a job finishes, "the summaries for all steps in a job are grouped together into a single job summary and are shown on the workflow run summary page." Monito writes a compact summary there — scenario results with links straight to the evidence — so the failure and its proof live on the run page, one click from the PR.
And the evidence is the part that makes a red check trustworthy instead of just alarming. Every run is a Monito Session: a screenshot timeline of what the agent did, the network log, console errors, and the agent's step-by-step reasoning at each judgment call. When the check goes red on "checkout," you don't get a stack trace pointing at a selector — you get the screenshot of the moment the total rendered as NaN, the failed request next to it, and a sentence explaining what the agent expected instead. That's the difference between a failure you can act on in the PR and a failure you have to reproduce locally before you believe it.
Because the Scenarios are prompts rather than recorded selectors, the check also doesn't fail on the PR that merely renamed the submit button — which is the exact false-positive that gets scripted preview suites disabled within a month of being introduced. A gate is only as valuable as its trustworthiness, and a gate that goes red on refactors trains people to ignore red. Playwright vs AI testing covers where each approach earns its keep; the one-line version is that the check nobody trusts is worse than no check, because it also blocks the merge button while being ignored.
The honest limits
Things this setup does not do, so you don't discover them at 3am:
It tests the code in the PR, not your production infrastructure. The agent runs against whatever URL you point it at — usually a stable staging environment — which has your frontend and your functions but not your production data volume, your real CDN and cache config, or your long-running migrations. A green PR check is not permission to skip the production smoke test; it's a different check catching different bugs. Run both.
It needs a reachable URL. This post assumes a stable staging environment the agent can hit on every PR. If instead you want to test the exact code in each PR on its own ephemeral deployment, that's the harder and more valuable version — and it has real wrinkles (the URL doesn't exist until the build finishes, and deployment protection will lock the agent out). That's a different post: AI QA on Vercel preview deployments handles it end to end.
Shared test data will bite you. If two concurrent PR runs both create test@example.com against the same staging database, one of them fails on a duplicate that isn't a bug. Generate unique per-run data — a fresh email per run — and have a cleanup story, or your gate develops a flake that has nothing to do with your code.
A gate is a subset by design. The PR check is the fast, load-bearing few. The broad coverage — long-tail flows, edge cases, the stuff you check weekly not per-PR — belongs in a scheduled Suite Run or a nightly job, not in the thing standing between a developer and the merge button. Keep the gate cheap so it stays fast, and put the breadth where slowness doesn't cost anyone a context-switch.
Wire it up
Set this up against your stable staging URL first, confirm it's green, and only then mark it a required check — you don't want the first thing your team sees from the new gate to be a false red on an unrelated PR.
The one-time setup is two browser sign-ins:
Then generate the workflow — either from the dashboard (open your Project, choose Automations → GitHub Actions, which produces the OIDC workflow shown above, the one that triggers on pull_request and push), or have your coding agent run the documented monito ci plan → apply → verify flow from the CI guide. Either path leaves you with .github/workflows/monito.yml and a repository-scoped trust binding, and no API key stored in GitHub.
Then the Scenario to gate on — the cheapest one that catches the most, a signup that has to actually reach the signed-in state:
Save it as a Test Scenario, commit .github/workflows/monito.yml, then flip on Require status checks to pass before merging and mark the Monito check required. Now a PR that breaks signup can't merge — not because someone remembered to look, but because the button is off. Your first run is free; point it at staging, open a throwaway PR, and watch a red check hold the line.