40 Questions Product Managers Should Ask Before Trusting AI
When you feed user research into ChatGPT or ask Dovetail AI to summarise patterns, you are trusting a machine to decide what matters. Your job is to know which customer problems will shape your product. These questions help you keep that responsibility.
These are suggestions. Use the ones that fit your situation.
Questions to ask when AI summarises your user research
1When Dovetail AI highlighted these three customer pain points, did it show me the original quotes or just a summary of them?
2If I read the raw interview transcript myself, would I agree with how the AI has grouped these insights, or would I see a different pattern?
3Which research participants did this AI summary pay the most attention to, and which ones disappeared into the background?
4The AI flagged this as a minor issue. Did it do that because it truly is minor, or because only one or two people mentioned it in different language?
5Has the AI smoothed over a contradiction in what users said, when that contradiction itself is the insight I need?
6When I look at the research the AI said was most important, can I remember the person who said it and why they said it?
7If this AI summary had said the opposite, how would I know the difference? What would make me question it?
8Are there any moments in the research where a user changed their mind or contradicted themselves that the AI glossed over?
9Did the AI find patterns because they are real in the data, or because they fit a pattern the model recognises easily?
10What would I have learned from this research if I had read it without AI help, and what am I missing now?
Questions to ask when AI ranks your product priorities
11When Jira AI suggested this ranking, what inputs did it use? Can I see which factors it weighted and which it ignored?
12This feature scored high. Is that because it will genuinely change user behaviour, or because it fits cleanly into the scoring system?
13The AI put this user problem lower because it affects fewer people. Have I confirmed that fewer people truly care about it, or did fewer people happen to mention it in research?
14If I prioritised the opposite way to what the AI suggests, what would break in the product?
15Does this ranking depend on any numbers I fed in that I am not certain about? Which ones?
16The AI suggested we defer this work. Was that because it is low impact, or because it cannot be measured easily?
17Have I talked to the people who would actually build these features in the order the AI suggests? What do they think?
18Is there a feature the AI ranked low that would stop a customer from using the product at all, even though it is not urgent right now?
19What customer segment did this prioritisation ignore? Would they agree with this ranking?
20If this ranking came from a colleague instead of an AI, what questions would I ask them to test it?
Questions to ask when building your roadmap from AI suggestions
21When Claude generated this roadmap, did it start from customer needs or from what I already told it was possible?
22This roadmap assumes we can build X, Y, and Z. Have I checked with the team whether that is realistic, or am I trusting the AI's guess?
23If I talk to customers about this roadmap, will they recognise their problems in it, or will they say we are solving for something different?
24The AI suggested we add this feature in Q2. Did it do that because it makes sense strategically, or because it fits a pattern like 'add one large feature per quarter'?
25Is there a customer problem in my research that this roadmap does not address? Why not?
26This roadmap looks logical when I read it. But did that logic come from the AI making things sound consistent, even if they are not connected to what users need?
27Have I sensed a shift in what customers care about that this AI roadmap might have missed because it was built from older data?
28If this roadmap failed to move our key metrics, would I know why, or would it be hard to tell what went wrong?
29Who in my organisation did the AI not hear from when building this roadmap? Should they have been included?
30When I present this roadmap to customers or stakeholders, which parts should I be cautious about because they came from an AI and not from direct conversation?
Questions to ask about your actual decision making
31In the last month, did I make a product decision based on an AI summary where I did not also read the original research myself?
32When was the last time I changed my mind because a customer told me something directly, rather than something an AI said about what customers want?
33If I had to explain to a customer why I prioritised their competitor's feature over theirs, what would I say, and would they accept that reason?
34Am I spending less time in actual customer conversations than I did before I started using AI research tools?
35The AI told me something was important. Did I test that by bringing it up with customers without suggesting the answer?
36Have I noticed that my roadmap now sounds more like what the AI tools I use tend to suggest, rather than what I believed before?
37If my AI research tool broke tomorrow, would my next roadmap be significantly different, or would it look roughly the same?
38Can I name a specific person who uses my product and what their main problem is? Or has the AI summary replaced that relationship?
39In a meeting, have I defended a decision by saying 'the data shows' when I really mean 'the AI said the data shows'?
40What would I learn if I talked directly to five customers this week about their needs, that my AI tools have not already told me?
How to use these questions
When Dovetail or Claude gives you a summary, treat it as a starting point. Read at least 20 percent of the raw data yourself. You will catch patterns the AI missed.
Before you act on an AI ranking in Jira, ask the person who will build it whether they agree. Their doubt is data.
Keep a list of insights from direct customer conversations that surprised you. Compare it monthly to what your AI tools flagged as important. The gaps are your blind spots.
When presenting a roadmap to stakeholders, say where it came from. Do not let 'the data suggests' become code for 'an AI thought this was reasonable'.
Schedule one hour each week to read raw research or talk to customers without running it through an AI first. This is how you stay close to the real problem.