All writing

Meta will send that webhook again

6 min read

An ordering agent that processes the same WhatsApp message twice bills a customer twice. Retries are not an edge case, they are the contract you signed.

I am building Storvey, a platform that lets Nigerian wholesale distributors take orders over WhatsApp through an AI agent. A customer messages the business number, an agent reads the message, checks the real catalog, and writes a real order into a real database.

That last part is what makes webhook delivery a money problem rather than a plumbing problem. If a chatbot answers a question twice, the customer is mildly annoyed. If an ordering agent processes the same message twice, someone receives forty crates of malt instead of twenty, and someone is owed a refund.

Meta retries webhooks. This is documented behaviour, not a fault. Any design that assumes exactly-once delivery is already broken.

Two different failures wearing the same coat

When I started writing the endpoint I thought I had one problem: do not process a message twice. I actually had two, and they feed each other.

The first is the obvious one. A retry arrives, you process it, the order doubles. The second is the cause of the first: if your endpoint is slow to acknowledge, the platform assumes delivery failed and retries. An agent turn involves a model call and several database round-trips. That is seconds. Acknowledging after all that work is what summons the retry you are then trying to survive.

So the fix is not one mechanism, it is two. Acknowledge fast so retries are rare, and be idempotent so the retries that still happen are harmless.

Acknowledge before you think

The endpoint does the smallest amount of work that is still correct, then returns. Verify the signature, claim the message, hand the real work to a background task, respond. The model call happens after the response has already gone out.

Signature verification has a detail worth stating plainly, because it is easy to get wrong in a way that appears to work. The HMAC must be computed over the raw request body, before any JSON parsing. If you parse the payload and re-serialise it to check the signature, you have changed the bytes: key order, whitespace, unicode escaping. The signature will not match, and the natural next move is to weaken the check until it passes. Read the raw bytes first, verify, then parse.

Claim the event, do not just record it

Idempotency here is a table with a uniqueness constraint on the event identifier. Before doing anything, insert a row. If the insert succeeds, this event is mine and I process it. If it fails because the row already exists, this is a redelivery, and I stop.

def claim_event(external_id: str, payload: dict) -> bool:
    """Persist the event BEFORE processing. Returns False if this
    event was already claimed, i.e. the platform is retrying a
    delivery we already have."""
    if not external_id:
        return False
    try:
        db.table("webhook_events").insert({
            "source": "meta",
            "external_id": external_id,   # unique constraint
            "payload": payload,
        }).execute()
        return True
    except Exception:
        return False
The shape of it. The constraint does the work, not the application logic.

The ordering matters more than the mechanism. Claim before processing, not after. If you process first and record afterwards, a crash in between leaves you with work that happened and no evidence it happened, and the retry does it all again. Claiming first means the worst case is an event marked as handled that was not fully handled, which is a recoverable state you can find and replay. The other way round, you lose money silently.

It is also worth noticing what the uniqueness constraint buys you over an application-level check. A read-then-write is a race: two concurrent deliveries both read nothing, both write, both process. The database constraint collapses that race into one winner and one exception, with no coordination code.

The same pattern, pointed at money

Webhooks do not only arrive twice. They also fail to arrive at all. A customer pays through Paystack, the confirmation webhook goes missing, and now the customer is looking at a balance that says they still owe.

The recovery is a reconciliation path. When a customer says they have paid and the balance disagrees, the agent asks Paystack directly rather than believing either the customer or its own records. If Paystack confirms a successful payment we never recorded, the service records it then, and tells the customer honestly that we had missed it. If Paystack has no successful payment, it says so plainly instead of inventing a reassurance.

The important part is that this repair goes through the same claim constraint the webhook uses. Two paths can now record a payment: the webhook and the reconciliation. Whichever arrives first claims the reference, and the second one finds the row already there and does nothing. A payment cannot be recorded twice regardless of which path sees it first, and I did not have to write any logic to coordinate them.

How I know it works

Two of the tests exist entirely to pin this behaviour down, and their names say what they protect:

  • test_duplicate_meta_message_processed_once, which feeds the same delivery in twice and asserts one order exists at the end
  • test_missed_webhook_is_self_healed_once, which simulates a lost payment webhook and asserts the reconciliation repairs it exactly once, not twice

There is a whole category of bug that only appears under conditions you cannot reliably reproduce by clicking around: the retry, the timeout, the crash between two writes. Tests are the only place most people will ever see those conditions before production does.

Storvey is not live to customers yet. The WhatsApp Business number is still in Meta review. So the honest version of this post is not that these mechanisms saved me during an incident. It is that I would rather write them while the cost is an afternoon than after a customer has been charged twice and I am reading logs at 2am trying to work out how many.

StorveyReliabilityFastAPIWebhooks

Got a project in mind?

Need a site or product shipped, or an existing one fixed up? Send me your goal and timeline.

Get in touch