40 Questions Data Analysts Should Ask Before Trusting AI Outputs
When AI generates your analysis, you become responsible for logic you did not write. These questions help you verify the reasoning before your stakeholders act on the findings.
These are suggestions. Use the ones that fit your situation.
1Did the AI include or exclude any rows based on assumptions I did not state?
2What date range did the AI use, and does it match the business period I need?
3If the AI filtered for 'active customers', what rule did it apply and is it the same rule my organisation uses?
4Did the AI handle null values by removing them, filling them, or ignoring them, and is that correct for this metric?
5When the AI joined tables, which key did it use and could that key produce duplicates?
6Did the AI apply any aggregation before the join that might have changed the row count?
7If the AI selected 'top 10' or 'top N' items, what was the sort order and did I ask for that?
8Are there any known data quality issues in the source table that the AI should have flagged?
9Did the AI assume a 1-to-1 relationship between tables when a 1-to-many relationship exists?
10When the AI ran the query, was it working from fresh data or cached results from earlier?
Questions About Statistical Logic and Calculations
11If the AI calculated an average, did it use mean, median, or mode, and is that the right choice for this data?
12When the AI showed a percentage change, did it use the correct baseline year or period?
13Did the AI use a moving average or a simple average, and was that choice explicit or hidden?
14If the AI flagged a trend as significant, what threshold or test did it use to decide?
15When the AI calculated year-on-year growth, did it account for missing months or incomplete data in either year?
16Did the AI apply any weighting to the values, or did it treat all rows as equal?
17If the AI split data into groups, how many records are in the smallest group and is that sample size reliable?
18When the AI calculated a ratio or rate, what denominator did it use and is it the standard denominator in your organisation?
19Did the AI annualise figures that were only partial-year, and was the method disclosed?
20If the AI identified an outlier, did it explain the threshold or just flag values that look unusual?
Questions About Visualisation Choices and Presentation
21Why did the AI choose a bar chart instead of a line chart, or a scatter plot instead of a table?
22What are the axis scales on this chart and could a different range make the pattern appear more or less dramatic?
23If the AI sorted categories in a specific order, is that alphabetical, by value, or by my stated preference?
24Did the AI truncate the axis to start at a value other than zero, and does that misrepresent the magnitude of change?
25When the AI colour-coded segments, what threshold determines green versus red, and is it a meaningful threshold?
26Does the chart show all the data, or did the AI filter or aggregate it without telling you?
27If the AI combined multiple metrics into one chart, are they on the same scale or does the visualisation need a secondary axis?
28Did the AI add a trendline or forecast line, and what statistical method did it use to generate it?
29When Tableau AI or Databricks AI suggested a chart type, did you verify it against your organisation's reporting standards?
30Are the labels on this chart clear enough that a stakeholder will understand what they are looking at without your explanation?
Questions About Assumptions and What You Did Not Ask
31Did the AI make assumptions about what 'customer segment' means or what counts as 'active' in your business?
32If the AI generated a summary statement, did it state the conditions or caveats that apply to that finding?
33When you asked the AI to find 'why sales dropped', did it identify actual causation or just correlation?
34Did the AI assume a linear relationship when a non-linear one might exist in your data?
35If the AI said 'there are no anomalies', does that mean it looked for them or simply did not check?
36When the AI provided a number without context, did you ask what the previous period was or what the target is?
37Did the AI account for seasonality, and if so, what historical period did it use to define normal?
38If the finding seems surprising, did you ask the AI to show its working or just accept the output?
39When the AI said 'based on available data', what data was it actually allowed to see?
40Did you verify the SQL or query logic that the AI wrote, or did you assume Code Interpreter got it right?
How to use these questions
Always run the AI-generated SQL against a small sample first. Check the row count and spot a few values by hand before trusting the full result.
When stakeholders ask for a finding 'by end of day', explicitly tell them you need two hours to verify the logic. Insight laundering happens when speed overrides scrutiny.
If you have not written SQL in three months because your AI tool does it for you, schedule time this week to write one query from scratch. Your statistical instinct atrophies when you stop practising.
Keep a record of one finding per month that the AI got wrong or misrepresented. Review it quarterly to stay aware of the types of mistakes your tools tend to make.
Before you share an AI-generated chart with stakeholders, ask yourself: would I stake my credibility on this output? If the answer is no, do not send it.