What happened
In November 2022, Jake Moffatt has just lost his grandmother. He asks Air Canada's chatbot how bereavement fares work. The chatbot tells him he can buy a ticket at the regular price and then claim a partial refund under the bereavement fare, as long as he does so within 90 days of the ticket being issued.
He buys his Vancouver–Toronto ticket at full price. When he asks for the refund, Air Canada refuses: its actual policy does not allow claims after travel. The 90-day rule only existed in the chatbot's answer.
Before British Columbia's Civil Resolution Tribunal, Air Canada argues that the chatbot is "a separate legal entity that is responsible for its own actions". The tribunal rejects the argument: the company is responsible for all the information on its website, whether it comes from a static page or a chatbot, and it did not take reasonable care to make its chatbot accurate. Air Canada is ordered to refund the difference, CAD 650.88, plus fees.
Why it happened
The amount is trivial; the precedent is not. A conversational agent speaking on the company's behalf binds the company. And a language model does not "know" it may not invent a rule: it produces the most plausible, most helpful answer. To a grieving person, the most helpful answer is a solution, even one that does not exist.
Nothing stopped the chatbot from stating a commercial policy that appeared in no validated document. This behaviour is everywhere: the agent promises. A goodwill gesture, a deadline, an exception, a rule.
The failure family
In our catalogue, this is the "Words" family: any agent that can write to a customer can make a commitment the company never decided. It does not depend on the business: an airline, an online shop or a bank are exposed in the same way.
The safeguard that would have prevented it
- Answers about policies (refunds, cancellations, guarantees) come only from a validated knowledge base, cited in the answer. No source, no answer: the agent hands over to a human.
- Any answer containing a commitment (amount, deadline, exception) is detected and sent for approval, or rephrased as "an adviser will confirm".
- Regular testing with customers in strong emotional situations, precisely when the agent most wants to help.
What we test
In a Crashmire crash test, a simulated customer insists, tells a hard story, invokes a rule that does not exist. We watch whether the agent confirms, promises or invents, and how many times over several runs, on each model.
And your agent, what would it do?
The crash test puts your agent in these situations, dozens of times, on several models. You get the figures, the conversations where it gave in, and the fix plan.