The agent is not allowed to guess

The agent behind Storvey takes wholesale orders over WhatsApp. It has tools: look up a price, check stock, create an order, generate a payment link, escalate to a human. Those tools touch inventory and money, which changes what a mistake costs.

Most writing about LLM guardrails is about stopping a model from saying something. My problem was the opposite. I had to stop a model from doing something, and the thing it most wanted to do was be helpful about a request it did not fully understand.

The rule that shaped everything

Anything the model says is a draft. Anything the model does goes through code that is allowed to say no.

Every guarantee that actually matters lives in a function, not in the system prompt. The prompt still matters, it shapes tone and the ordinary path, but a prompt is a strong suggestion to a probabilistic system. If a rule must hold every single time, it belongs somewhere deterministic.

This sounds obvious written down. It is surprisingly easy to violate, because when you find a failure the fastest fix is always to add a sentence to the prompt telling the model not to do that. It usually works, in testing, most of the time. That is exactly the trap.

Refusing to guess

A customer messages: send the usual come my shop. There is a version of this product that looks up their order history, infers the usual, and places it. It would feel magical roughly nine times out of ten. The tenth time it ships the wrong thing to a business that now has to argue with a machine about it.

So a vague order gets a clarifying question and nothing else. The test asserts the behaviour rather than the wording: no order-creating tool was called, no order id was produced, the orders table has exactly as many rows as before. It is called test_vague_order_is_not_guessed, and it fails if the agent gets clever.

The same rule covers products we do not stock. The agent cannot invent a price, because prices only come from a catalog lookup scoped to that business. If the product is not there, there is nothing to quote. An order mixing valid and unknown items creates nothing at all rather than partially succeeding, because a half-placed order is worse than a rejected one: the customer believes they ordered five things.

Where the model does not get a vote

Some rules are business rules with money attached, and those are enforced before a tool does its work, not by asking the model to remember them.

• An order beyond a customer's credit limit is blocked and escalated to the owner, whatever the conversation sounded like
• An order beyond available stock reports the real available quantity instead of accepting the number requested
• Payment amounts come from the payment provider's server response, never from the model's understanding of the conversation
• A payment link charges what is outstanding, not the order total, so a partly-paid order cannot be charged twice over

That last one is the kind of detail you only find by writing the test. The natural implementation charges the order total, and it looks correct until a customer pays half and then asks for a link.

The backstop I am most attached to

There is one failure I decided had to be impossible rather than unlikely: the agent tells a customer that a human will get back to them, and no human ever finds out.

The model has an escalate tool and is instructed to use it. Usually it does. But the failure mode is specific and quiet: it writes a warm, reassuring sentence promising follow-up, and does not call the tool. Nothing errors. The customer waits for a call that was never queued.

So after the reply is generated and before it is sent, there is a deterministic check in code. If the reply promises a human follow-up and the model did not escalate, the service escalates on its behalf.

# Deterministic backstop, in code not prompt: if the reply
# PROMISES a human follow-up but the model never called
# escalate_to_human, escalate on its behalf.
if not ctx.escalated and promises_followup(reply):
    dispatch_tool(ctx, "escalate_to_human", {
        "reason": "Auto-escalated: the reply promised the "
                  "customer a human follow-up",
    })

Two things about this are worth pulling out. First, the phrase list covers Nigerian Pidgin as well as English, because customers code-switch and so does the model replying to them. A backstop that only recognises make I get back to you in standard English would miss make I confirm am entirely, and the customer it strands would be a real customer.

Second, the list is deliberately over-broad, and that is a deliberate asymmetry rather than sloppiness. A false positive costs a human about ten seconds to dismiss an item in a queue. A false negative costs a customer who was promised a callback and never got one. When the two error directions cost that differently, tuning for precision is the wrong instinct.

Deriving instead of asking

One smaller decision I keep being glad about. Every conversation is logged with an intent, and the obvious way to get one is a second model call asking what just happened. Instead the intent is derived deterministically from which tools actually ran during the turn.

It is cheaper and faster, but the real reason is that it cannot be wrong in a new way. A model asked to classify a conversation can hallucinate a category. A function that reads a list of executed tool calls is reporting a fact. When you are building the audit trail you will later use to work out what went wrong, that trail should not itself be a guess.

None of this makes the agent safe in an absolute sense. It makes the blast radius of being wrong small, bounded, and visible, which is the most I think anyone can honestly claim about putting a language model near a database.