How to test a webhook receiver: duplicates, out-of-order events, and replays

Search "how to test a webhook receiver" and every result hands you a tunnel. Webhook.site, ngrok, Svix Play, Beeceptor — they give you a URL that catches an incoming request and shows you the headers and body. That's genuinely useful for about ten minutes, while you're first wiring the thing up. It answers exactly one question: did a payload arrive, and what did it look like?

It answers none of the questions that page you at 3am. Did the receiver charge the customer twice when Stripe delivered the same event twice? Did it flip the subscription to active and then canceled because the two events arrived out of order? Did it wedge itself into a retry storm because it returned a 500 on a payload it had already processed? A tunnel that echoes the request tells you nothing about any of that, because all of that lives in your system's state after the request — not in the request itself.

This is the playbook we'd run against a webhook receiver, written as plain-English prompts. The reframe that makes it tractable: you stop testing the payload and start testing the consequence. The signature check you can unit-test. Whether your product ended up in the right state after a duplicate, a reorder, or a failure is a browser-and-database question, and that's where a QA agent earns its keep.

What "testing the receiver" actually means

A webhook receiver is a tiny, deceptively hostile piece of software. It's an unauthenticated-by-default POST endpoint that a third party calls on their schedule, with at-least-once delivery, no ordering guarantee, and a retry policy you don't control. Every one of those properties is a test case, and none of them show up in the happy-path payload.

Here's the honest scope of a real check, in rough order of how often each one ships broken:

1. Signature verification. The receiver must reject any request that isn't genuinely from the provider. Stripe signs every event and is explicit that you need the raw body to verify it: "Stripe requires the raw body of the request to perform signature verification. If you're using a framework, make sure it doesn't manipulate the raw body. Any manipulation to the raw body of the request causes the verification to fail." The classic bug is a body-parser middleware that JSON-parses and re-serializes before your handler ever runs, silently breaking verification for everyone.
2. Idempotency on duplicate delivery. Providers deliver at least once, which means sometimes more than once. Stripe says it plainly: "Webhook endpoints might occasionally receive the same event more than once. You can guard against duplicated event receipts by logging the event IDs you've processed, and then not processing already-logged events." If your receiver provisions a license or sends an email on each delivery, a duplicate is a double-charge or a double-email.
3. Out-of-order events. "Stripe doesn't guarantee the delivery of events in the order that they're generated," and their own example is a subscription that emits customer.subscription.created, invoice.created, and invoice.paid — which can land in any order. A receiver that assumes created always arrives before paid will build state on a foundation that isn't there yet.
4. Retry behavior on failure. When your endpoint returns a non-2xx, providers retry. Stripe retries for up to three days with exponential backoff; GitHub expects a 2xx within 10 seconds or it "terminates the connection and considers the delivery a failure." A receiver that does slow work inline before responding turns every slow request into a duplicate delivery.
5. The 4xx-vs-5xx decision. A malformed or already-processed event should usually get a 2xx (accept and ignore) or a deliberate 4xx (don't retry this, it'll never succeed). Returning a 5xx for a permanent problem is how you manufacture an infinite retry loop against yourself.

Notice that only the first item is about the request. The other four are about what your system did, observed after the fact. That's the part the tunnel tools structurally cannot see, and the part that actually breaks in production.

The bug everyone ships once: the duplicate that isn't idempotent

Before the prompts, the failure to look for first, because nearly every integration ships it at least once: a handler that does exactly the right thing exactly once, and the wrong thing twice.

The naive handler reads the event, does the work — provision the seat, mark the invoice paid, send the receipt — and returns 200. It passes every test you write against a single delivery. Then one day a network blip makes the provider retry a delivery it already made, or your own slow response causes a redelivery, and the same event runs your side effects a second time. Now there are two seats, two receipts, two rows.

The fix is well-known (dedupe on the event ID before doing side effects), but the reason it keeps happening is that the duplicate path is invisible on the happy path. You have to deliberately deliver the same event twice and then go look at whether the effect happened once or twice. A person can do this by hand a few times and then never again. An agent can do it on every deploy, because "go look at whether the effect happened once or twice" is a plain-English instruction, not a selector.

If the concept of asserting on end-state rather than on wire format is new to you, why AI QA agents find bugs your scripts miss is the background; this post is that idea aimed at one specific surface.

Setup: what you drive, what the provider drives

Be clear-eyed about the division of labor, because it's what keeps these tests honest.

A browser agent can't forge an HMAC-signed POST from Stripe — nor should it try. What it can do is drive the two ends that matter: it can trigger real deliveries by acting in your app (complete a test-mode checkout and a genuine checkout.session.completed fires), and it can read the resulting state anywhere it surfaces in your product — the orders list, the billing page, the admin panel, the activity log. The provider's own tooling handles the deliveries a user can't naturally cause: the Stripe CLI's stripe trigger fires a specific event, and stripe events resend re-delivers one you've already seen, which is your duplicate and your out-of-order generator.

So the pattern for each test below is: put the app in a known state, cause the delivery condition (by acting in the browser, or with one CLI command), then let the agent assert on what the product did. You'll need a staging environment pointed at the provider's test/sandbox mode, a test account the agent can log into, and the receiver wired to real (test-mode) provider events. Pass credentials through your Test Scenario config rather than baking them into prompt text — the same way the OAuth callback playbook handles provider secrets.

Prompt 1: the happy path, exactly once

Boring first. If the effect doesn't happen once, nothing else is worth testing.

Go to https://staging.yourapp.com and log in with
test@example.com / Password123!

Complete a test-mode checkout for the "Pro" plan using the test
card 4242 4242 4242 4242, any future expiry, any CVC.

After checkout, go to the Billing page and the Orders list and
verify that:
- The account shows exactly ONE active "Pro" subscription
- The Orders list shows exactly ONE order for this purchase
- The order status reads "Paid" (not "Pending" or "Processing")
- No duplicate or ghost rows appear for the same purchase

Report how long after checkout the status became "Paid", and
flag anything that looks off — including things I didn't ask about.

That timing report is quietly the point: the gap between checkout and "Paid" is your webhook round-trip. If it's instant, you might be optimistically updating the UI before the webhook confirms — which is its own bug when the webhook later fails.

Prompt 2: the duplicate delivery

The one that catches the double-charge. This needs one CLI command alongside the agent.

Precondition: a completed test-mode purchase exists for
test@example.com (from Prompt 1 or set up fresh).

I will re-deliver the checkout.session.completed event a second
time using `stripe events resend <event_id>` while you watch.

Go to https://staging.yourapp.com, log in as test@example.com,
and open the Billing page and the Orders list.

After the event is re-delivered, verify that:
- There is still exactly ONE active subscription (not two)
- There is still exactly ONE order row (no duplicate)
- The customer was not emailed a second receipt (check the
  in-app notifications/activity log if visible)
- Nothing about the account state changed from the single-delivery
  version

Flag any evidence the event was processed twice.

If the second delivery produces a second seat, a second row, or a second receipt, you've found the idempotency gap — and you found it by looking at effects, which is the only place it's visible.

Prompt 3: out-of-order arrival

Stripe's own docs warn that a subscription lifecycle can arrive in any order. Test the assumption directly.

Using the Stripe CLI, I will deliver two related events in the
"wrong" order for a new test customer — invoice.paid BEFORE
customer.subscription.created.

Go to https://staging.yourapp.com and log in as the admin at
admin@example.com / AdminPass123!

Open the customer's account in the admin panel and verify that,
once both events have been delivered:
- The subscription exists and is in the correct active/paid state
- There is no "orphaned payment" with no subscription attached
- There is no subscription stuck in an incomplete/pending state
  that never reconciled
- The account is in the SAME final state it would reach if the
  events had arrived in order

Describe the final state in detail so I can compare it to the
in-order run.

A receiver that assumes ordering will usually leave one of two fingerprints: an orphaned payment (the paid event had nowhere to attach) or a permanently-pending subscription (the created event overwrote the paid state). Both are invisible until you inspect the account.

Prompt 4: the retry storm and the 5xx trap

The failure mode that takes down your own endpoint. GitHub's guidance is that a receiver should respond with a 2xx within 10 seconds and do heavy work on a background queue; violate that and every slow request becomes a retried — and therefore duplicated — request.

Go to https://staging.yourapp.com and log in as admin@example.com.

I will deliver a webhook event that your receiver currently
mishandles (an event type for a resource that was already deleted,
which should be safely ignored).

After delivery, verify in the admin panel and activity log that:
- The event was acknowledged and ignored cleanly (no error state
  on the account, no half-applied change)
- The system did NOT enter a repeated-processing loop — the
  activity log should show the event handled once, not the same
  event reappearing every few minutes

If you see the same event reprocessing on a loop, that is a 5xx
being returned for a permanent condition, and it is the bug.
Report the exact repetition interval if it happens.

The repetition interval, if it shows up, is the provider's backoff schedule — a dead giveaway that your receiver is returning a retryable status for something that will never succeed on retry.

Reading the results

Every failed Test Run gives you the full Monito Session: the screenshot timeline, the console output, the network log, and the agent's step-by-step reasoning. For webhook bugs the money shot is the state comparison across runs — the single-delivery account versus the duplicate-delivery account, side by side. When the two differ, you don't just know that the receiver is non-idempotent; you can see what it double-applied.

Save each prompt as a Test Scenario in your Project and run the set against staging on every release, or wire it into CI on your preview deploys so a receiver regression fails the PR that caused it. Each run is roughly 8–13 credits (about $0.08–$0.13), and because the prompts assert on product state rather than on selectors, they survive a full rewrite of your webhook handler.

The consumer-side-in-staging approach here is the same one we used for testing rate limiting without hammering production: induce the hostile condition deliberately, in a safe environment, and assert on what your system does about it.

The one to run first

If you only wire up one, make it the duplicate — it's the highest-severity, most-commonly-shipped bug of the five. Here's the version to paste in:

Go to https://staging.yourapp.com and log in with
test@example.com / Password123!

Complete a test-mode checkout for the "Pro" plan using the test
card 4242 4242 4242 4242. Note the resulting order and subscription.

Then I will re-deliver the same checkout webhook a second time.

After the re-delivery, verify the account still shows exactly ONE
active subscription and exactly ONE order — no duplicates, no
second receipt, no changed state. Report exactly what, if anything,
the second delivery changed, and flag anything else that looks
broken.

Your first run is free — point it at a staging checkout and find out whether your receiver can survive being told the same thing twice.