40 Questions Customer Success Managers Should Ask Before Trusting AI Churn and Retention Predictions
Your best customers often show signs of unhappiness in conversations before any metric shifts. When Gainsight flags churn risk or Einstein rates a customer as stable, you need to know whether the AI has seen what you have seen. These questions help you stay the expert in your own customer relationships.
These are suggestions. Use the ones that fit your situation.
1When Gainsight AI flags this customer as high churn risk, what specific behaviour changed and over what time period?
2Did the usage decline happen gradually or as a sudden drop, and do I remember a business change at this customer's company that might explain it?
3Is the AI comparing this customer's usage to their own baseline or to a cohort average, and does that distinction matter for this customer type?
4What if the usage drop is intentional because they finished a project or seasonal cycle, and not a sign they are leaving?
5Has my relationship contact at this account changed in the past three months, and if so, does the AI know that?
6When I listened to Gong recordings from this customer's calls, did their tone or urgency shift before the churn flag appeared?
7Did the AI flag churn risk because of a contract renewal date approaching, or because of actual behaviour changes that suggest they will not renew?
8Are there customers who show identical usage patterns but are actually very healthy, and if so, why should I trust this pattern for this customer?
9Has this customer's exec sponsor been in health check calls, or is the account flying blind without executive visibility?
10What would it look like if this customer was preparing to expand rather than leave, and does the AI distinguish between those two scenarios?
Questions About AI-Generated Customer Communications
11If I send this ChatGPT-generated check-in email instead of calling, what relationship signals am I missing?
12Does this AI-templated message reference anything specific about this customer's industry, use case, or stated goals, or does it sound like it could go to anyone?
13Have I talked to this customer in the past month, and if not, should an email from me really come from an AI?
14When Intercom AI suggests a response to a customer's support message, does it account for the tone I have built with this person over two years?
15Is this the moment to automate, or is this the moment when the customer needs to know I noticed something and cared enough to respond myself?
16Does the AI-suggested communication assume the customer is behind on adoption, when actually they are about to expand into a new department?
17If I were the customer receiving this message, would I know if it came from a real person who tracks my account or from a system sending batch messages?
18Am I using AI to write to customers because it saves time, and if so, at what cost to the relationship?
19Does this AI message get me closer to understanding why the customer has gone quiet, or does it just make contact without real listening?
20When was the last time I sat on a call with this customer, and should that gap be filled with an email or a conversation?
Questions About AI-Reported Metrics and Success Scores
21When Salesforce Einstein shows this customer has a health score of 72, what are the three behaviours that number is based on?
22Is the health score improving because the customer is actually getting value, or because they are logging in more out of frustration trying to solve a problem?
23If the AI reports strong usage across the account, but all the activity is concentrated in one user who is about to leave, does the score reflect that risk?
24What does the AI not measure that I know matters for this customer's success, and am I blind to that because the dashboard looks green?
25Has the AI metric system been validated against customers who actually churned to see if it would have predicted their departure?
26If a customer's NPS score is declining but their usage score is stable, which one is telling me the real story?
27When the dashboard says this customer is on track, have I verified that they are actually on track to achieve their stated business outcome?
28Does the success metric I am looking at track activity or value, and is there a difference between those two things for this customer?
29If three months ago I would have flagged this customer as at risk based on what I knew, but the AI says they are healthy now, what changed?
30Am I making decisions about which customers to invest time in based on AI scores, and if so, which relationships am I neglecting as a result?
Questions About Your Own Judgement and Early Warning Instincts
31When was the last time I had an instinct that a customer was in trouble before any data showed it, and have I documented what I noticed?
32If I stopped trusting my own ability to hear tension in a customer conversation, what would I lose?
33In the past year, how many customers did I save because I knew them well enough to notice something was off before the AI would have caught it?
34When Gong AI transcribes a call, am I actually listening to the recording, or am I just skimming the AI summary?
35Do I still remember how to read a customer's hesitation in an email, or have I outsourced that judgement to sentiment analysis?
36If I had to predict churn for my top 10 accounts without access to any AI tool, how confident would I be, and should that confidence be lower than it was two years ago?
37When a customer tells me something in conversation that contradicts what the AI dashboard shows, whose account do I trust first?
38What decisions am I making on my own versus what decisions am I now waiting for the AI to recommend, and is that shift making me better or worse at my job?
39Have I noticed myself becoming slower to reach out to customers because I am waiting for an AI alert, or faster because I am responding to alerts?
40If the AI tool disappeared tomorrow, would I still know how to read the room with your customers, or would I have to relearn that skill?
How to use these questions
Before you act on any churn flag, ask yourself: would I have noticed this myself in conversation, and if not, why should I change my action based on the AI noticing it for me?
When you write a customer message, draft it yourself first, then ask whether AI would improve it or just make it faster. Faster is not always better for retention.
Keep a list of three customers each quarter who showed early warning signs in conversation before usage metrics changed. Use that list to pressure test whether your instincts are still sharp.
If your health score dashboard ever shows your at-risk customers as stable, stop trusting the numbers and call those customers. Do not wait for the system to agree with you.
Block time on your calendar each week to have one unscheduled conversation with a customer based on genuine curiosity, not on an AI recommendation. Protect that space.