40 Questions Economistss Should Ask Before Trusting AI
AI tools like ChatGPT and Claude can generate plausible-looking regression models and confidence intervals in seconds, but they often embed assumptions you have not verified. Your judgement about what drives an economy remains more valuable than any model, and asking the right questions before accepting an AI output protects both your analysis and the policy decisions that depend on it.
These are suggestions. Use the ones that fit your situation.
1When Claude or ChatGPT suggests a specification for your econometric model, can you trace back to economic theory that justifies including each variable, or have you accepted it because it improved the fit?
2Has the AI tool made assumptions about lag structures, functional forms, or causal direction that you would reject if they came from a junior colleague without explanation?
3Do you know which economic relationships in your model are identified by theory versus which ones the AI identified by pattern matching in historical data?
4When Bloomberg AI or Stata AI suggests dropping a variable to improve model performance, have you considered whether that variable matters for the policy question you are answering, even if it adds noise?
5If you ran this model using 1980s data instead of recent data, would the AI have suggested the same variables and functional form?
6Has the AI incorporated structural breaks or regime changes that your knowledge of the economy tells you matter, or does it treat all historical periods as equivalent?
7When the AI produces a demand model, can you explain the transmission mechanism for each included variable, or does it work as a black box that you cannot defend in peer review?
8Have you tested whether the relationships the AI model estimates are stable across different subsets of your data, or have you accepted them because they fit the full sample?
9Does the AI model respect known constraints from economic theory such as adding-up restrictions or homogeneity conditions, or have you let the data speak without those guardrails?
10If you needed to explain this model to a central bank's monetary policy committee, which assumptions would make them pause and ask you to reconsider?
On Forecasts and Confidence
11When Claude generates a 95 percent confidence interval for next quarter's GDP growth, do you know whether that interval accounts for model uncertainty, parameter uncertainty, and genuine economic volatility, or is it merely a statistical product?
12Has the AI tool produced a forecast for a period of structural change by extrapolating a model fitted to stable historical conditions, and have you flagged this as unsuitable for policy use?
13Do you know how many forecast errors the AI-assisted model made during the 2008 financial crisis, the Covid shock, or other tail events, or have you only examined recent performance?
14When Perplexity or ChatGPT presents a forecast range for inflation, does that range widen as the forecast horizon lengthens to reflect genuine uncertainty, or does it narrow because the AI has not properly modelled long-run uncertainty?
15If your AI-generated forecast proved wrong in six months, could you explain to your director why the model failed, or would you simply retrain it on new data?
16Have you compared your AI model's forecast uncertainty to the actual disagreement among expert forecasters, or have you assumed the statistical measure of confidence matches real-world uncertainty?
17Does the AI acknowledge that its forecast confidence is conditional on the assumptions built into the model, or does it present confidence intervals as if they reflect absolute certainty about the future?
18When you generated this forecast using AI, did you update your priors about what could go wrong, or have you let the model's narrow confidence bands reduce your professional skepticism?
19If a policy maker acts on your forecast and economic conditions change unexpectedly, are you prepared to explain that forecasts are inherently uncertain, not that the AI failed?
20Have you tested whether the AI model can forecast through structural breaks by retraining it on pre-break data and testing it on post-break data, or have you only tested within-sample fit?
On Policy Analysis and Recommendations
21When ChatGPT or Claude suggests a policy instrument, does it rest on economic theory you believe in, or have you adopted it because it emerged from pattern matching in policy documents?
22Has the AI model estimated the policy response by fitting historical relationships that may not hold when policymakers change their behaviour in response to new shocks?
23If the AI recommends a particular tax rate, interest rate path, or spending level, have you examined the distributional consequences across income groups, regions, or sectors that the model may not have emphasised?
24Do you know whether the AI-assisted policy analysis accounts for behavioural responses from firms and households, or does it assume static reactions to policy changes?
25When an AI tool ranks policy options by model-predicted outcomes, have you considered second-order effects or political economy constraints that the model does not capture?
26Has the AI model been tested on historical policy episodes that resembled your current policy question, or is it applying patterns from unrelated contexts?
27If your government adopted the policy recommendation from your AI-assisted analysis and results diverged from the forecast, would you be able to explain why without retreating to model caveats?
28Does the AI analysis acknowledge trade-offs between competing policy objectives such as growth versus inflation control, or does it optimise for a single metric?
29Have you stress-tested the policy recommendation by asking what happens if key model parameters are half their estimated values or double, or have you accepted the point estimates?
30When presenting the AI-assisted policy analysis to decision makers, are you clear about which conclusions rest on economic theory you endorse and which rest on statistical patterns the model found?
On Your Own Analytical Judgement
31If the AI output contradicts your economic intuition about how this market or policy channel works, have you investigated why, or have you updated your intuition to match the model?
32When you use Stata AI or Bloomberg AI to accelerate your analysis, have you maintained the same critical review standard you would apply to a model built without AI assistance?
33Can you identify a recent analysis where you caught the AI making a mistake before presenting it, or does the AI's authority cause you to skip critical steps?
34Do you still spend time reading economic theory and historical cases, or has the speed of AI tools crowded out the thinking time that produces good economic judgement?
35When Claude generates multiple model specifications, do you choose the best one according to your own criteria, or have you defaulted to whichever specification the AI ranked first?
36Have you recently rejected an AI-suggested approach because it violated economic reasoning, even though it would have improved model fit?
37Do you know your own blind spots in economic reasoning, and have you designed your use of AI tools to protect against those specific weaknesses?
38When you present economic analysis to policymakers or your peers, do you distinguish between findings that rest on your professional judgement versus findings that came from AI pattern matching?
39Have you tested yourself by manually working through an economic problem before using AI, and then comparing your reasoning to the AI output?
40If AI tools became unavailable tomorrow, could you still produce credible economic analysis, or have you outsourced too much of your analytical thinking?
How to use these questions
Before trusting any AI-generated coefficient, ask yourself whether you could defend the underlying economic mechanism to a sceptical peer who knows your field.
Document your assumptions separately from your AI outputs. If the AI embedded an assumption you did not explicitly choose, you have not done the analysis yourself.
Use AI to accelerate execution of analysis you have already designed, not to design the analysis for you. Sketch your model on paper first.
When an AI tool gives you a narrow confidence interval or a point forecast, imagine that interval is twice as wide. Does the policy recommendation still hold?
Treat AI-generated outputs as a first draft that requires your economic judgement before they become analysis. No AI output should go directly into a policy brief.