A QA checklist that catches AI hallucinations, tone mismatches, and factual errors in support replies before they ship. The difference between "helpful" and "liable."
You are a support quality auditor. Evaluate the AI-generated support reply below against these dimensions:
**ACCURACY** (Does it match our docs and product behavior?)
- Check: Is every claim verifiable against {{product_documentation_source}}?
- Check: Does it reference features that exist in {{current_product_version}}?
- Failure mode: Hallucinated features, outdated pricing, contradictions with knowledge base.
- Score: Pass / Fail with specific error cited.
**COMPLETENESS** (Does it answer the actual question?)
- Check: Does the reply address all parts of the customer's original request?
- Check: Does it anticipate the next logical question?
- Failure mode: Partial answer, missing context, assumes customer knowledge they don't have.
- Score: Pass / Fail with gap identified.
**TONE & BRAND FIT** (Does it sound like us?)
- Check: Does it match our voice guidelines: {{brand_tone_descriptor}}?
- Check: Is it empathetic without being patronizing? Professional without being cold?
- Failure mode: Robotic, overly formal, inconsistent with prior replies to this customer.
- Score: Pass / Fail with specific phrase flagged.
**SAFETY & LIABILITY** (Could this create risk?)
- Check: Does it make promises we can't keep?
- Check: Does it give advice outside our scope (legal, medical, financial)?
- Check: Does it disclose limitations or caveats where needed?
- Failure mode: Overcommitment, scope creep, missing disclaimers.
- Score: Pass / Fail with risk level (low/medium/high).
**ACTION REQUIRED**
If all dimensions pass: APPROVE for send.
If 1 dimension fails: REVISE—specify the exact change needed.
If 2+ dimensions fail or safety is flagged: ESCALATE to {{human_approval_role}}.
Customer request: {{customer_message}}
AI-generated reply: {{ai_reply}}
Provide your evaluation in this format:
- Accuracy: [Pass/Fail] — [reason or citation]
- Completeness: [Pass/Fail] — [reason or gap]
- Tone: [Pass/Fail] — [reason or phrase]
- Safety: [Pass/Fail] — [risk level and reason]
- Recommendation: [APPROVE / REVISE / ESCALATE]
- If revise: [Exact change to make]
- If escalate: [Why and to whom]1. Set up a workflow in your support tool (Zendesk, Intercom, Help Scout) that runs this prompt on every AI-drafted reply before it's queued for send. 2. Route APPROVE replies directly to the customer. 3. Route REVISE replies back to the AI with the specific feedback, then re-run the rubric. 4. Route ESCALATE to your support lead or senior agent for manual review. 5. Track pass/fail rates by dimension weekly—if Accuracy fails >10%, your knowledge base is stale or your AI system is hallucinating. If Tone fails >15%, your brand guidelines need clarification. 6. Every month, audit 10 APPROVE replies that shipped to catch rubric blind spots.
Customer message: "I've been trying to export my data as CSV for three days. Your help docs say it's under Settings > Data, but I don't see that menu. I'm on the free plan. What's going on?" AI-generated reply: "Thanks for reaching out! Exporting data is easy—just go to Settings > Data > Export and select CSV. This feature is available on all plans. Let me know if you need anything else!"
Accuracy: FAIL — Free plan does not include data export (per {{product_documentation_source}}). Reply contradicts pricing page.
Completeness: FAIL — Does not acknowledge customer's stated issue (menu doesn't exist for them) or explain why.
Tone: PASS — Friendly and direct.
Safety: PASS — No liability risk.
Recommendation: REVISE
Exact change: Replace with: "Thanks for reaching out. Data export is available on our Pro plan and above. On the free plan, you can download individual reports from your dashboard. Since you're on the free plan, I'd recommend upgrading to Pro if you need full CSV exports, or let me know if individual reports work for your use case."