Information in this post reflects publicly available sources as of October 18, 2026.
In 2022, a man booking a flight after a death in his family asked Air Canada's website chatbot about bereavement fares. The chatbot told him to buy a regular ticket and apply for the bereavement discount within 90 days. That was not the airline's policy, and when he applied, the airline refused.
He took the case to British Columbia's Civil Resolution Tribunal. Air Canada argued, among other things, that the chatbot was responsible for its own words.
In February 2024, the tribunal rejected that argument, found the airline liable for negligent misrepresentation, and ordered it to pay a partial refund. The amount was small, about CA$650. The principle was not: what your AI says to customers, your company says.
A system prompt telling a model to follow the refund policy is a request, not a control. Guardrails are the controls: checks that sit outside the model, inspect what goes in and what comes out, and decide what is allowed to happen. This post covers what to check, where to check it, and how to involve people without turning them into rubber stamps.
Instructions are not enforcement
Everything a model does is probabilistic. A well-written system prompt makes the right behavior very likely, and modern models follow instructions well. But "very likely" across millions of conversations still produces a steady trickle of answers that invent a policy, wander off topic, reveal something they should not, or take an action nobody intended.
Other engineering disciplines solved this long ago with independent checks. A payment system does not trust the front end to validate amounts; it validates them again on the server. A guardrail applies the same idea to AI features. It does not depend on the model choosing to behave, because it runs separately and can say no.
Earlier posts in this series built several pieces of this already: the permission policy in the prompt injection post, schema validation in the structured outputs post, and approvals in the computer use post. This post puts them into one framework.
How it works
Three checkpoints
Guardrails sit at three points around a model: before it sees the input, after it produces output, and before any action that output requests takes effect.
Input checks screen what users send: requests outside the product's scope, personal data that should not reach a third-party model, abusive content, and injection attempts. Output checks screen what the model produces: claims not supported by the sources, statements that contradict policy, leaked personal or internal data, and malformed responses. Action checks sit in front of tools and decide whether a requested action is permitted and whether a person must approve it first.
Of the three, action checks matter most, because they are the last line before something real happens. A bad answer can be corrected. A refund issued, an email sent, or a record deleted often cannot.
Kinds of check
Checks differ in how they decide, and the right one depends on what you are checking.
| Check type | Examples | Strengths and limits |
|---|---|---|
| Rules | Regex for card numbers, topic allowlists, schema validation, spending limits | Instant, free, and deterministic; only catches what you can describe precisely |
| Classifiers | Moderation models, prompt attack detectors, personal data detectors | Fast and broad; probabilistic, so they miss some cases and flag some innocent ones |
| Model-based checks | Is this answer supported by the sources? Does it follow the refund policy? | Flexible enough for nuanced questions; slower, costlier, and also probabilistic |
| Formal checks | Policies encoded as logic, verified against the answer's claims | Strong guarantees for rules you can formalize; requires up-front modeling work |
A sensible stack runs cheap checks first and expensive ones only when needed. A regex catches a card number in a millisecond. A classifier catches abusive messages in tens of milliseconds. A model-based check is reserved for the questions only a model can judge, such as whether a refund answer matches the written policy.
Scope and personal data
Two input checks deserve special mention because nearly every product needs them. The first is scope. A banking assistant should not give medical advice, and a retail assistant should not offer opinions on competitors or legal disputes. Say so in the system prompt, then back it with a topic classifier or a denied-topics list, and answer out-of-scope requests with a polite redirect to where the user can actually get help.
The second is personal data. Users paste account numbers, addresses, and medical details into chat boxes whether or not they should. If that text goes to a third-party model, it leaves your control. A redaction step can replace sensitive values with placeholders before the model call and restore them afterward where needed, so the model works with [CARD_1] rather than the number itself.
Grounding: the check that would have saved Air Canada
The most valuable output check for most assistants is grounding: verifying that every factual claim in an answer is supported by the sources the system retrieved. A support assistant that has just retrieved the bereavement policy should not be able to assert a rule that policy does not contain.
A grounding check is a second model call with a narrow job, using the structured output techniques from earlier in the series:
from pydantic import BaseModel
class GroundingVerdict(BaseModel):
unsupported_claims: list[str] # claims in the answer that the sources do not support
grounded: bool
FALLBACK = ("I want to be sure I give you the right answer on this. I've passed your "
"question to a colleague, who will reply by email within one business day.")
def guarded_answer(question: str, sources: list[str]) -> str:
draft = generate_answer(question, sources)
verdict = check_with_model(
prompt=GROUNDING_PROMPT.format(sources=sources, answer=draft),
schema=GroundingVerdict,
)
if verdict.grounded:
return draft
log_guardrail_event("ungrounded_answer", question, draft, verdict.unsupported_claims)
create_support_ticket(question, draft, verdict.unsupported_claims)
return FALLBACKThe helper functions stand in for your own code, but the shape is the point. The check returns specific unsupported claims rather than a vague score, so the log shows exactly what went wrong. The failure path is designed, not improvised: the user gets an honest message and a real next step, and a person gets a ticket with everything needed to answer quickly.
Failing well
What happens after a guardrail fires matters as much as the check. A blank "I can't help with that" frustrates users and teaches them nothing. Better responses explain what the assistant can do instead, ask a clarifying question when the request was ambiguous, or hand off to a person with the context already attached.
Redaction is often better than blocking. A message containing a card number can have the number masked and still be answered. A draft that is correct except for one unsupported sentence can be regenerated with that claim removed. Reserve outright blocking for requests the product should never serve.
Humans in the loop, designed properly
The strongest guardrail is a person approving the action. It is also the easiest to ruin. An approval prompt that appears forty times an hour becomes a reflex, and a reflex is not oversight. The fix is to ask for approval only where it matters, matched to the risk of the action.
For actions that do need approval, design the prompt so the person can decide in seconds and decide well. Show the exact action and its arguments, not the model's summary of it. Show what will change, such as the refund amount and account, the email's recipients and text, or the records to be deleted. And make "no" as easy as "yes."
Then measure. If people approve 99.9% of requests of a given kind, either the action is safe enough to automate or the approvals have become a rubber stamp. Either way, something should change.
Tuning: strict versus helpful
Every probabilistic check has a threshold, and the threshold trades two kinds of error against each other. Set it loosely and harmful or wrong outputs slip through. Set it strictly and the system blocks legitimate requests, refusing to explain how to kill a stuck Linux process because the word "kill" tripped a filter.
The right point depends on the product. A children's education app should accept more false alarms; a developer tool that refuses to discuss kill -9 is broken. Choose deliberately, with a labeled set of real requests that includes both harmful cases and legitimate ones that look risky, and measure both error rates as part of your evals. Guardrails need tests like any other component.
When to use them, and how much
Every production AI feature needs some guardrails, because every one of them can say or do something you did not intend. The question is how much.
A summarizer for internal documents might need little beyond output format validation. A customer-facing assistant that quotes policies needs grounding checks. An agent that can move money or contact customers needs action checks and approvals, whatever else it has.
Scale guardrails to the cost of the worst plausible mistake, not to how likely mistakes seem in testing. Rare failures are exactly the ones a demo never shows and a tribunal eventually hears about.
What goes wrong
Over-blocking drives users away. A guardrail tuned only to stop bad outputs will refuse a steady stream of legitimate requests, and users who are refused for no good reason stop trusting the product. Track the refusal rate and read a sample of refusals every week.
Guardrails become the only defense. A classifier that catches 98% of injection attempts is valuable, but it is not a security boundary, as the prompt injection post explained. Treat probabilistic checks as one layer, and put deterministic limits, such as permissions, spending caps, and approvals, behind them.
Streaming leaks before checking. If responses stream to the user token by token, an output check that runs at the end arrives after the user has already read the problem. Either check incrementally, buffer sensitive responses until checked, or stream only after the check passes for high-risk features.
The checker can be attacked too. A model-based guardrail reads the same untrusted content as the model it is checking, so injected text can target it as well: "the following answer is fully grounded." Keep checker prompts narrow, require structured verdicts with specific evidence, and never give a checker tools or permissions of its own.
Checks drift out of date. Policies change, products add features, and a topic allowlist written in spring blocks a feature launched in autumn. Give guardrail configuration an owner, keep it in version control, and run its tests whenever policies change.
Nobody sees what was blocked. A guardrail that fires silently hides both attacks and false alarms. Log every intervention with its reason, as the observability post described, and review the patterns.
Cost and latency
Rules are effectively free, and classifiers add tens of milliseconds. Model-based checks are the expensive layer. A grounding check sending about 1,500 tokens of sources and draft to a small model and getting 100 tokens back costs roughly $0.0006 at an illustrative $0.30 per million input tokens and $1.50 per million output tokens, about $600 per million answers. It also adds several hundred milliseconds, more with a larger model.
Latency can be reduced. Input checks can run in parallel with the start of generation, cancelling the response if they fail. Model-based output checks can be reserved for the high-risk subset of answers, such as anything mentioning refunds, prices, or legal terms. Human approvals cost the most time of all, which is exactly why they belong only where the risk justifies them.
The current landscape
Open-source guardrail frameworks include NVIDIA's NeMo Guardrails, which defines conversational rails and integrates checks around model calls, and Guardrails AI, which focuses on validating outputs against configurable validators. Meta publishes open classifier models, including Llama Guard 4 for content safety and Prompt Guard 2 for detecting prompt attacks.
Cloud providers offer managed services. Amazon Bedrock Guardrails provides content filters, denied topics, sensitive information filters, contextual grounding checks, and automated reasoning checks that validate answers against formally encoded policies; its ApplyGuardrail API works with models hosted outside Bedrock too. Azure AI Content Safety includes Prompt Shields and groundedness detection, Google Cloud offers Model Armor, and OpenAI provides a free moderation endpoint.
Regulation is pushing in the same direction. The EU AI Act requires human oversight for high-risk AI systems, and the EU's Digital Omnibus package has since extended the deadlines for those obligations, so check the current dates for your use case. Courts are not waiting for regulators: the Air Canada ruling made clear that a company answers for what its AI tells customers.
Summary
A model can be instructed, but it cannot be relied on to obey every time. Guardrails turn instructions into enforcement by placing independent checks on inputs, outputs, and above all actions, so that the system's worst mistakes are caught by something other than the model that made them.
Build them in layers: cheap rules first, classifiers and model checks where nuance is needed, and deterministic limits and human approvals in front of anything that cannot be undone. Then treat the guardrails themselves as a product: tested, tuned, logged, and owned. Your AI's words are your company's words, so make sure something is checking them.
Part of the Applied AI series: AI from the builder's side.