Scope the tasks the bot can handle
Choose repetitive, low-risk questions with approved answers. Define escalation paths for payments, complaints, sensitive information and cases that lack a matching source. Ask support staff which questions are safe to handle automatically.
Define what the bot can answer and when it should hand off
Review a sample of real support reasons and identify repeated questions your organisation can answer from approved documents. Start with bounded topics such as delivery windows, basic account navigation or policy explanations. Set rules for escalation, identity verification and exceptions. Give customers a way to reach a person without completing an unnecessary conversation.
- Classify the requestIdentify whether it is public, private or exceptional
- Answer within scopeUse approved content or a permitted service
- Hand off and reviewSend uncertain or sensitive requests to the right team
Connect a governed knowledge base
Organise approved help articles, assign owners, and remove obsolete policies. Have the bot cite the relevant guidance where possible. Limit what user details are retained and test for situations that require an agent to take over.
Build the knowledge and action layers deliberately
Assign owners for policy documents, revision dates and conflicting instructions. Retrieve relevant passages from those sources for each answer and test whether the bot can show where its answer came from. Handle sensitive account data only through properly authorised services, with access controls outside the model. Start with read-only or low-risk operations and require clear confirmation before an approved external action.
Evaluate before general release
Use representative conversations and intentionally difficult cases. Review answer accuracy, successful handoffs and customer sentiment. Start with a limited rollout and give support staff a clear way to disable an unreliable workflow.
Evaluate normal requests, edge cases and handoff quality
Create a test set with ambiguous questions, unsupported requests, contradictory documents and attempts to redirect the bot outside its job. Review correctness, escalation and privacy separately. Check whether agents receive enough context after handoff. Pilot with limited traffic while monitoring repeat contacts, agent feedback and genuinely resolved requests.
A support bot for order and return questions
A retailer lets its chatbot explain public return policies and status terminology. The bot sends order changes and exceptions to a logged-in support route. In the pilot, agents discover that a policy answer confused two delivery regions, so the editor separates policies and adds a regional test question.
Set a review routine for cases where the bot does not know the answer. Classify whether the failure came from missing source material, a bad retrieval result, an incorrect policy interpretation or an unavailable tool. Give an owner a way to correct the underlying information and add a regression example. If you expand the bot to another language or support market, retest policies and handoff paths instead of assuming the first rollout covers those cases.
| Task | Default path | Verification |
|---|---|---|
| Public policy question | Answer from an approved source | Compare the reply with current policy |
| Account-specific request | Authenticated service or human handoff | Check permissions and minimise data |
| Exception or uncertain answer | Escalate with conversation context | Have agents score the handoff |
Common questions
Should I start by fine-tuning a model?
Start with a clean knowledge source and a clear evaluation set. Then see whether retrieval, prompting, workflow changes or other modelling work addresses observed failures.
Is a high bot containment rate enough?
Review satisfaction, repeat contact and correctness too. Containment can rise because customers could not find an obvious escalation path.
How do I decide if the bot is ready to launch?
Define acceptable limits for incorrect answers and failed handoffs, review a representative test set, and confirm support can take over when the system fails. Start with limited traffic.
What to check next
- KnowledgeUse approved, dated help content
- HandoffTest common and risky escalation cases
- ReleaseRoll out gradually and monitor corrections




