Natural language test automation: what your English actually compiles to
Natural language test automation means two very different things: English that compiles to a fixed script, or English an agent interprets at run time. Here's how each works under the hood, and when plain English holds or leaks.
Natural language test automation: what your English actually compiles to
"Write tests in plain English" is the most repeated promise in QA tooling right now, and it hides a fork in the road that most vendor pages never mention. There are two fundamentally different technologies wearing the "natural language test automation" label, and they behave nothing alike when your product changes, when a check gets fuzzy, or when a test fails at 2am and you need to know why.
This is the neutral explainer we couldn't find when we went looking: what natural language test automation actually is, what your English turns into under the hood in each model, and — because every abstraction leaks — exactly where plain English holds and where it doesn't.
The fork: English as syntax vs. English as intent
Model 1: English compiles to a program. In tools like testRigor, plain English is a real, documented command language: click "Submit", check that page contains "Welcome, Peter!". You author test cases in that language, the platform resolves each command to an element and an action, and the resulting sequence runs the same way every time. The English is friendlier than JavaScript, but it is still source code — deterministic, reviewable, versionable, and yours to maintain. We wrote a full comparison of that model in our testRigor piece; the one-line summary is that you're still authoring and gardening a test catalog, just in a much better notation.
Model 2: English is interpreted at run time. In the agent model, your English never becomes a stored program at all. You write intent — "test the signup flow, try invalid emails and a weak password, make sure the errors are sane" — and an AI agent reads the rendered page in a real browser, decides what to do next, does it, and looks at the result before choosing the next step. Nothing compiles. Nothing is stored except the prompt and the evidence. Each run is a fresh interpretation of the same intent.
The distinction sounds academic until you rename a button. In model 1, some command somewhere referenced that button, and either self-healing absorbs the change or the test breaks — we covered what self-healing actually fixes (short version: selectors, not meaning). In model 2 there is no reference to break, because the instruction was never about a button — it was about signing up.
What a prompt compiles to in the agent model
"The agent figures it out" is not an explanation, so here is the actual pipeline when Monito executes natural language. You save your English as a Test Scenario — a reusable prompt attached to a Project. When it runs, the Agent executor opens a real Chromium browser and starts the loop: read the rendered page, pick the next action toward your intent, act, observe.
The part most people don't expect: the agent's conclusions aren't free-form English either. As it works, it calls a small set of structured tools, and those calls are what your results are made of:
assert_visible— the agent asserts that something specific is visible on the page, and that assertion is recorded as a checkable event, not a vibe.capture_checkpoint— a named milestone with a screenshot, so the run's story has fixed points you can trust.report_bug— a structured bug report: what was expected, what happened, steps to reproduce, evidence attached.mark_result— the explicit pass/fail verdict with reasoning.
So natural language in, structured events out. Every browser action and every tool call lands in a Monito Session — screenshots, network log, console errors, the agent's reasoning — which is the artifact you actually review. If you'd rather consume it programmatically, the AI agents guide shows how to pull runs and session events as JSON from the CLI.
This is the answer to the fair objection "how do I trust a test I can't read?" You don't read the test — there isn't one. You read the evidence, which is more than most scripted suites give you.
Where natural language holds
Plain English is genuinely the better interface in a few specific situations, and it's worth being precise about them.
Intent-level regression. "Sign up, log in, reset the password, confirm the emails make sense" survives redesigns, copy changes, and restructured forms, because none of those change the intent. This is the property that makes the agent model cheap to own: the prompt has no maintenance surface. There are no selectors in it to update.
Fuzzy judgment. "Flag anything that looks broken on this page" is a one-line instruction to an agent and nearly unwritable in any command language, because a command language needs the check enumerated element by element. The checks you most want a human tester for — does this error message make sense? does the layout survive 375px? — are exactly the ones that don't compile.
The person writing isn't an engineer. A founder, a PM, or a support lead who can describe a user flow can write a Test Scenario. That's not a marketing line; it's a direct consequence of the input being intent rather than code. AI QA testing explained walks through what a first run looks like end to end.
Where natural language leaks
Every abstraction leaks, and it's better to know the holes up front than to find them in production.
Ambiguity is a bug in your prompt. "Verify the totals are correct" — which totals, correct relative to what? An agent will make a reasonable interpretation, and reasonable is not the same as yours. The fix is the same discipline as writing a good bug report: name the data, state the expected outcome. "Add two items priced $10 and $15, verify the cart shows $25 before tax" leaks nothing.
Determinism. An agent reasons each run, so two runs of the same Scenario aren't byte-identical — same intent, possibly different steps. For most regression questions that's fine, and it's the price of adaptability. But if you need a CI gate that fails on exactly one condition and never flakes in either direction, a small scripted assertion is the honest tool, and the case for and against owning that script deserves its own decision. Many teams run both: agent for breadth, a thin pinned suite for the two flows that must be exact.
Precision assertions. "The chart renders correctly" is not something English can pin down to the pixel. Numeric tolerances, pixel-exact rendering, byte-level API contracts — spell them out in the prompt if they can be observed on the page ("the response time shown should be under 500ms"), or use a purpose-built tool if they can't.
Auditable test cases. If a compliance regime wants a versioned catalog of deterministic test cases, model 1 tools are built for exactly that world, and an agent's reasoned runs are the wrong artifact. Choose accordingly.
Writing prompts that don't leak
The practical skill of natural language test automation is not prompt "engineering" — it's the same skill as briefing a new QA hire. Three habits cover most of it:
State outcomes, not clicks. "Complete checkout with the test card and confirm the order number appears" beats a ten-step click list. You hired the agent to find the steps; over-specifying them just re-creates the brittleness you left scripts to escape.
Name your data. Test accounts, card numbers, exact strings you expect to see. Every concrete value in the prompt is one less interpretation the agent has to make.
Ask for judgment explicitly. The closing line of a good prompt is standing permission to go beyond the script: "report anything that looks broken, slow, or confusing, even if the main flow passes." That line is where agents earn their keep — it's why they find bugs your scripts miss.
One underrated consequence of prompts-as-tests: review gets easy. A Test Scenario is a short paragraph of English, so when a teammate changes one, the diff is readable by anyone — the PM can review the QA change, which is not a sentence anyone has said about a Playwright PR. The prompt is small enough to reason about as a whole, and there's no imported helper file three directories away silently changing what the test does.
Try the compile step yourself
The fastest way to understand the difference between English-as-syntax and English-as-intent is to run one prompt that contains both a precise check and a fuzzy one, and watch how the agent handles each. Point a Test Scenario at your staging environment:
Step 2 is a precise assertion the agent will record with assert_visible. Step 3 doesn't compile to commands in any language — and it's where your next real bug is hiding. Your first run is free; Monito is $99/mo after that (Enterprise $129/mo), with a typical full run costing 8–13 credits, roughly $0.08–$0.13.