Chatbot Deployment Checklist

Decide whether to expand, tune, or pause

This checklist helps a small community team decide whether a narrowly scoped chatbot pilot is ready to expand, needs another test cycle, or should pause. The default is to keep the pilot limited until every critical safeguard has a named owner and evidence from your own context. Use the canonical interactive checklist as the working copy, then explore the Safer Admin & AI Workflows topic for workflow-mapping guidance.

Protect member information before testing

Do not send credentials, private records, analytics exports, screenshots containing private information, or identifiable sensitive records. Use fictional or deliberately sanitised test cases, confirm what the selected tool stores, and keep safeguarding, complaints, eligibility, account access, and other decisions about people with an accountable human. OWASP's current GenAI guidance treats sensitive-information disclosure and prompt injection as application risks; a chatbot launch checklist therefore needs data boundaries as well as answer-quality checks.

How to use this checklist

Review the nine areas with the people who own community policy, member support, source content, privacy, and the pilot channel. Mark an item complete only when you can point to a document, configured control, named owner, or reviewed test result. Record gaps rather than averaging them away: one missing high-impact safeguard can be a reason to pause even when most boxes are checked. NIST's voluntary AI RMF and Playbook organise risk work around Govern, Map, Measure, and Manage; the checklist turns those ideas into a small-team launch review.

1. Scope and decision rights

Start with a bounded support job, such as answering approved opening-hours, membership, or event-access questions. Do not make "answer anything" the pilot scope.

List the in-scope intents and the questions the chatbot must decline or hand to a person.
Name the person who may approve scope changes and the person who can pause the pilot.
Write the member outcome you are trying to improve and the harm you must not increase.
Keep decisions about safeguarding, complaints, eligibility, rights, or individual support with a human.

2. Approved knowledge and source ownership

A useful answer should be traceable to an approved source, not merely sound plausible.

List the exact FAQs, policies, service pages, and contact routes the pilot may use.
Remove conflicting or expired material before testing.
Assign an owner and review date to every source group.
Define what happens when a source is missing, ambiguous, or out of date.

3. Safety, privacy, and governance controls

Treat the chatbot as a workflow with inputs, permissions, logs, and human decisions, not as a standalone answer box.

Document prohibited input and output categories in language moderators can apply.
Restrict configuration, source editing, and transcript access by role.
Set a retention rule for test prompts, transcripts, and review notes.
Test prompt-injection attempts, requests for private information, and unsafe instructions.
Record who reviews incidents and how the pilot can be disabled.

4. Human handoff

A handoff is part of the service, not an exception hidden at the end of a failed conversation.

Give members a visible way to request a person without having to trick the chatbot.
Route by issue type and potential impact, using thresholds your team has defined and tested.
Pass the member intent, the sources used, and steps already attempted in the handoff summary.
Tell the member what will happen next without promising a response time the receiving team cannot meet.
Test the receiving queue, named owner, and out-of-hours path end to end.

5. Response quality and accessibility

Review answers for usefulness in the setting where members will read them, including on a small screen or with assistive technology.

Check factual accuracy against the approved source and require links where a member should verify details.
Use plain language, short steps, and the community's established tone.
Make uncertainty and refusal clear without blaming the member.
Check link labels, heading order, keyboard access, zoom, and screen-reader output in the actual interface.
Include common wording variations, follow-up questions, and incomplete prompts in the test set.

6. Test evidence and failure handling

Build a small repeatable test set for every in-scope intent, plus edge cases. Classify each miss as a source gap, retrieval problem, unsafe answer, unclear wording, or handoff failure. A critical safety or privacy failure is not cancelled out by a high overall pass rate.

Keep the expected answer or expected handoff beside each test case.
Have a person other than the configurator review a sample.
Record failures with an owner, corrective action, and retest result.
Retest the whole affected intent group after changing a source or rule.

7. Run a controlled pilot

Choose one channel or member segment where the team can observe failures and intervene. State the pilot start and review dates, staff the human route, and freeze scope while you collect comparable evidence.

Tell participants that they are using a limited chatbot pilot and that a human route remains available.
Monitor unresolved or high-impact conversations during the pilot window.
Review failures and source gaps at a cadence the named owners can sustain.
Stop or narrow the pilot when the agreed pause condition is reached.

8. Define measures from your baseline

Do not borrow a universal response-time, containment, satisfaction, or confidence threshold. Define measures from the service members currently receive, the risk of a wrong answer, the capacity of the human queue, and the smallest sample you can review responsibly. Google's SRE guidance treats service objectives as service-specific, stakeholder-approved targets that should be refined; use the same discipline here rather than treating a generic number as proof of readiness.

Measure the same in-scope intents, channel, and time window before and during the pilot.
Track answer correctness, appropriate handoff, member effort, response time, and unresolved cases separately.
Write an owner-approved target or guardrail for each measure before reviewing the result.
Pair summary rates with the underlying reviewed count and inspect high-impact failures individually.

Worked measurement example

Hypothetical example, not a benchmark: a team reviews 50 pilot conversations. Ten test cases should have reached a person, and eight did, so appropriate handoff is 8/10 for that sample. The team also records two missed handoffs individually because their impact matters more than the 80% summary. If its pre-pilot median first response for the same question set was 18 hours, it can compare the pilot result with that 18-hour baseline. Whether either result is acceptable remains the team's documented decision, informed by risk, member expectations, and available human cover.

9. Make the go, tune, or pause decision

Record one of three decisions and the evidence behind it. Expand only when every critical control passes, the reviewed pilot meets your named guardrails, and no unresolved high-impact failure remains. Tune when the use case is still appropriate but a source, wording, routing, or staffing gap has a credible owner and retest. Pause when the team cannot protect inputs, ground answers, provide a reliable human route, review failures, or name an accountable owner.

Two failure modes to watch

Failure mode 1: scope expands before intent quality is stable. The symptom is a rising cluster of wrong answers or escalations after new intents are added. Freeze additions, fix the highest-impact intent, and resume only after two consecutive review cycles meet the thresholds your team documented. Failure mode 2: escalation is treated as a fallback rather than a designed workflow. Members repeat context, requests reach the wrong queue, or internal targets are missed. Add a structured handoff summary, test the receiving queue, and alert the named owner when the team's queue-level guardrail is breached.

Use the checklist and map the next workflow

Open the interactive chatbot deployment checklist to record the review. If the pilot exposes an unclear handoff or ownership gap, use the workflow mapping worksheet to map that process before adding more automation. These are self-serve planning aids, not launch certification or professional advice.