40 Questions Healthcare Organisations Should Ask Before Trusting AI Diagnostic and Clinical Tools
When an AI system recommends a diagnosis or clinical action, your clinicians need to know whether that recommendation is safe to act on. These 40 questions help you build the judgement needed to use AI tools without letting them replace human accountability.
These are suggestions. Use the ones that fit your situation.
1When Epic AI flags a diagnostic possibility, does your system record which clinician reviewed the output and what they decided to do with it?
2If a patient is harmed after an AI recommendation was followed, can you retrieve the exact training data and decision logic that produced that recommendation?
3Does your clinical governance process require sign-off from a named clinician before any AI-assisted diagnosis becomes part of the patient's official record?
4Has your trust explicitly documented which clinical decisions can be made with AI support and which ones still require independent human assessment?
5When Google Health or Microsoft Azure Health makes a recommendation, do your protocols specify the level of evidence needed before a junior clinician can act on it alone?
6Does your AI implementation track how often clinicians override AI recommendations, and do you review those cases for patterns?
7Are your patient safety incident reports set up to flag cases where AI output was a factor in adverse outcomes?
8Does your trust have a process for communicating changes to AI tool accuracy or behaviour to all clinicians using them?
9When IBM Watson Health suggests treatment pathways, is there a documented fallback protocol if the clinician disagrees?
10Has your clinical governance lead reviewed the liability implications of each AI tool you are using, in writing?
Patient Safety and Clinical Reasoning
11Can a junior doctor in your trust explain why DeepMind AlphaFold made a particular protein structure prediction, or do they just accept the output?
12If an AI diagnostic tool is right 95 percent of the time, do your clinicians know what the 5 percent of errors look like?
13Does your training programme for new staff include scenarios where AI recommendations are confidently wrong?
14When Epic AI suggests a diagnosis, do your protocols require the clinician to generate at least one alternative hypothesis themselves?
15Are your senior clinicians still doing diagnostic reasoning work, or has AI pushed that work away from them and toward junior staff?
16Does your trust measure whether clinicians are building diagnostic skills at the same rate they did before AI implementation?
17When Google Health or Azure Health analyses imaging, is a consultant radiologist required to review the output before it shapes treatment decisions?
18Do your protocols specify which patient populations the AI tool was tested on, and whether your local population differs in ways that might change its accuracy?
19If Watson Health recommends a treatment that conflicts with local clinical guidelines, is that flag highlighted to the clinician?
20Has your trust documented the cognitive load on clinicians who must review and validate AI outputs during routine clinical work?
Transparency, Data, and Bias
21Does your vendor provide the demographic breakdown of patients in the training data for your diagnostic AI tools?
22If Epic AI performs better for some patient groups than others, have you documented which groups and why?
23Can you access a plain-language explanation of what features the AI tool is using to make its recommendations, or is it a black box?
24Does Microsoft Azure Health or Google Health flag when it is operating outside the distribution of data it was trained on?
25Have you tested your AI diagnostic tools on your own patient population to see if performance matches the vendor's published figures?
26When IBM Watson Health makes a recommendation, does your system show the confidence level and what that confidence is based on?
27Has your trust reviewed whether any of your AI tools were trained on data that includes documented historical biases in diagnosis or treatment?
28Do your clinicians know what happens if they submit a patient case that the AI tool has never seen before?
29Has your organisation negotiated contractual rights to audit the performance of Epic AI, Azure Health, or other tools on your own data?
30When DeepMind AlphaFold makes a prediction about protein structure, does your research team have access to confidence scores and uncertainty estimates?
The Therapeutic Relationship and Patient Trust
31Does your consent process tell patients when AI has been used in their diagnosis or treatment recommendation?
32If a patient asks whether their diagnosis came from a clinician or an AI tool, can your staff give a clear answer?
33Have you measured whether patients are less likely to trust treatment recommendations when they know AI was involved?
34Does your trust have a policy on whether clinicians can tell patients that an AI tool is infallible or more accurate than human judgment?
35When Epic AI assists with diagnostic reasoning, does the clinician spend the same amount of time talking with the patient about the diagnosis?
36Are your clinicians trained to explain to patients what AI did and did not do in their care?
37If a patient is diagnosed using Google Health or Microsoft Azure Health, does your organisation offer them the option of a second opinion from a human clinician?
38Has your trust considered whether AI-assisted care could discourage patients from raising concerns with clinicians?
39Do your patient information leaflets and consent forms mention the AI tools used in your diagnostic or treatment pathways?
40When Watson Health automates administrative decisions about care pathways, does your trust track whether patients feel they had adequate involvement in those decisions?
How to use these questions
Assign one senior clinician in your organisation ownership of AI governance. They should review every AI tool before it reaches routine use, and audit performance quarterly.
Test any new AI tool on 50 cases from your own patient population before you deploy it. Your patients may differ from the vendor's training data in ways that change accuracy.
Require written sign-off from a consultant-level clinician on every AI-assisted diagnosis that becomes part of the patient record. Document who reviewed it and when.
Train junior doctors on why an AI tool is wrong in specific cases, not just how to use it. Automation bias grows when staff never see the failure modes.
Audit your patient safety incident reports every month for cases where AI output was a factor. If you find patterns, change your protocols before the next patient is harmed.