A chatbot demo is easy to fall in love with. Someone types a question, the bot answers it well, and the room nods. Then it goes live, gets connected to a real order system, a real CRM, a real knowledge base that changes weekly, and within a month it's either quietly disabled or actively making things worse.
The model isn't usually what breaks. The wiring around it is.
The part everyone scopes
Ask a team to plan a chatbot project and they'll produce a plan for the parts that are fun to plan: which model to use, what tone the bot should have, what a good conversation looks like, maybe a retrieval setup so it can answer from a knowledge base instead of hallucinating. All of that is real work, and none of it is where projects actually die.
What doesn't show up on that plan is the list of systems the bot has to read from and write to once it's live: the order management system, the CRM, the ticketing queue, inventory, billing, auth. Each one has its own API, its own rate limits, its own auth model, its own definition of "up to date," and its own opinion about what happens when something goes wrong mid-request. A conversation design doc doesn't have a section for "what happens when the CRM times out during a live chat," so it usually doesn't get one until that happens for real, with a customer watching.
Two incidents worth studying
Two well-documented cases from 2024 make the integration gap concrete instead of abstract.
In February 2024, Air Canada's website chatbot told a customer he could apply for a bereavement fare after the fact, which contradicted the airline's actual policy. When the customer tried to claim the discount, Air Canada argued in front of Canada's Civil Resolution Tribunal that the chatbot was "a separate legal entity that is responsible for its own actions," and that the airline wasn't liable for what it said. The tribunal rejected that argument and ordered Air Canada to honor the bad information its own bot gave out. Nothing dramatic disconnected the bot from reality. It just had no reliable link back to the actual, current policy text, so it filled the gap with something that sounded plausible.
The second case is a security failure rather than a factual one. In December 2023, a car shopper testing a Chevrolet dealership's website chatbot told it to agree that a 2024 Chevy Tahoe would sell for one dollar, and to treat that as a legally binding offer with "no takesies backsies." The bot, built on a general-purpose LLM wired directly into the dealership's chat widget with no constraints on what it could commit to in writing, obliged. The screenshots spread fast enough that the dealership pulled the bot down soon after. Nothing about the model was defective; nobody had scoped what the bot was and wasn't allowed to agree to before pointing a public chat window at it.
Neither failure was a model quality problem. Both were integration and guardrail problems: no verified path back to ground truth, and no boundary on what counts as a real commitment.
The failure modes that don't show up in a demo
A few patterns repeat across the projects that stall after launch:
Stale retrieval. A knowledge base gets indexed once at launch. Then pricing changes, a policy gets updated, a product gets discontinued, and the bot keeps citing the old version because nobody built a re-indexing job. Just a one-time import script, running once and never again.
No handoff path. The bot handles the easy 80% of questions fine and has nothing graceful to do with the other 20%. Instead of routing to a human with the conversation context intact, it either loops the customer through the same three unhelpful answers or drops them with no path forward at all.
Unhandled tool failures. Modern chatbots built on models like GPT-4 or Claude typically use function calling (also called tool use) to query real systems instead of guessing. A tool definition might look like this:
{
"name": "get_order_status",
"description": "Look up the current status of a customer order by order ID",
"parameters": {
"type": "object",
"properties": {
"order_id": { "type": "string" }
},
"required": ["order_id"]
}
}
That works fine when the order system responds in 200ms with clean data. It works much worse when the order system is slow, returns a partial record, or is down for maintenance. Most integration plans only test the happy path, so the failure path ships untested. Nobody notices until a real order gets stuck mid-checkout.
No rate-limit or auth story. CRMs and helpdesk platforms enforce request caps and token expiry the bot has to respect. A design that assumes unlimited, always-authenticated access to a backend system will work in staging and start silently dropping requests the first time real traffic hits it.
Security exposure from wiring real data into a prompt. Once a bot can pull customer records, order details, or account data into context to answer a question, that data sits inside the same prompt an attacker might be trying to manipulate. A support bot that looks up any account from just an email address, with no verification step, hands an attacker a working path to real customer data. That gap rarely gets caught in a demo, because a demo never tries.
What actually needs scoping before day one
Chatbots aren't the problem here. The integration layer needs the same rigor as the conversation design, and it needs it before launch, not after the first bad headline.
A few concrete things worth doing before a bot touches real customers:
- List every backend system the bot will read from or write to, and get a straight answer from each system's owner about its rate limits, auth expiry, and what a degraded response looks like.
- Decide, in writing, what the bot is never allowed to agree to or commit to (pricing, contract terms, policy exceptions), and enforce that as a hard constraint, not a prompt instruction it can be talked out of.
- Build and test the failure path deliberately: kill a dependency in staging and watch what the bot does when a tool call times out or returns an error.
- Design the handoff to a human as a first-class feature, not a fallback bolted on later, with the full conversation history carried over so the customer doesn't have to repeat themselves.
- The retrieval index needs an owner and a refresh schedule. Treating it as a one-time export is how it goes stale by month two.
The bot itself is usually the easy part. Getting it to talk to real systems, reliably, under real load, with real consequences for getting it wrong: that's the project most teams never actually scope.
