What Makes an AI Tool Explainable to a Patient

An AI tool is explainable to a patient when that person can answer four plain questions without a technical background: what does this tool do, what does it not do, why did it suggest this for me, and who is responsible for the decision. Those four answers, in everyday language, matter more to a patient than any chart of model weights. A patient does not need to see inside the math. They need to know how the tool fits into their care and who stands behind it.

That distinction sits at the center of my work. As Head of Medical and Science on the EASY Diabetes program, helping develop AI clinical decision support for type 2 diabetes, one lesson kept returning. The explanation an engineer needs to trust a model is not the explanation a patient needs to trust their care. Both are real, and they are different jobs.

A short definition of patient-facing explainability

Patient-facing explainability is the ability of a person receiving care to understand, in their own terms, what an AI tool contributes to a decision about them and who remains accountable for that decision. It is not the same as model transparency, which describes how the system works on the inside.

Think of it like a lab result. Most people cannot describe the chemistry of a glucose assay, and they do not need to. What they need is the number, what it means for them, what to do next, and the knowledge that a clinician is reading it with them. Explainability for a patient lives at that level. It is about meaning, not mechanism.

Why technical explainability is the wrong tool for this job

Engineers explain models to each other so they can debug, audit, and improve them. They study which inputs moved a prediction, how the model behaves across groups, and where it fails. This work is essential. A tool that no one can inspect has no business near a patient.

But hand that same material to the person in the exam chair and it does very little. Knowing how much a single input nudged a risk score does not tell someone whether to worry, what to ask, or whether their clinician agrees. Technical detail can even mislead, because precision on a screen can feel like certainty about you, and those are not the same thing.

So the question is not how much detail we show. It is which questions we answer, shaped around what a person actually wants to know when their health is involved.

The four questions a patient should be able to answer

What does this tool do?

A patient should be able to say, in one sentence, what the tool is for. Maybe it estimates a risk over time, or flags a value that deserves a second look, or suggests a possible next step for the care team. The job of the tool should be stated as plainly as the job of a thermometer.

In decision support for diabetes, the honest framing is narrow and reassuring. The tool helps organize information and surface patterns so a person and their clinician can have a clearer conversation. Naming that modestly is more trustworthy than overselling it.

What does it not do?

This is the question most often skipped, and it may be the most important one. A patient deserves to know the edges of the tool. A risk estimator does not diagnose. A pattern detector does not decide treatment. None of them replaces clinical judgment about the whole person in the room.

Naming the limits is not a weakness to hide. It is part of what makes a tool safe to rely on. When a person knows what a tool will not do, they know when to lean on it and when to lean on their clinician instead.

Why did it suggest this for me?

People do not want a lecture on algorithms. They want the local reason. Why this, why now, why me. A useful answer points to things a patient recognizes from their own care, such as recent values or a trend over a few months. The aim is recognition, not revelation.

A good rule I have come to trust is that an explanation should let a patient sensibly disagree. If someone can look at the reason and say, that does not sound like my situation, the clinician now has something concrete to discuss. An explanation that cannot be questioned is not really an explanation.

Who is accountable?

Behind every suggestion there is a person and a process. A patient should know that an AI tool informs care, it does not own it, and that a clinician remains responsible for what happens next. Accountability is not a footnote. It turns a clever output into trustworthy care.

This is where regulation and clinical responsibility meet daily practice. My training in medical device regulation and clinical investigation keeps pointing at the same principle. A tool used in care has a defined intended use, known limits, and a human chain of responsibility, and the patient is entitled to know that chain exists.

What this looks like in practice

When we ran the EASY-1 trial across more than forty clinics with several hundred participants, the value of the tool was never the score by itself. It showed up in the conversation the score made possible. A clearer picture in front of a patient and a clinician at the same time tends to produce better questions, and better questions tend to produce better care.

That experience changed how I think about a good explanation. It is not a popup with a confidence number. It is a sentence a clinician can say out loud and a patient can repeat to a family member that evening. If it survives being retold at a kitchen table, it was written for the right audience.

There is a quieter benefit too. When a person understands what a tool is watching for, they often become a more active partner in their own monitoring. None of this is cause for alarm. These tools exist to catch patterns early, and early attention is almost always on your side.

A reasonable checklist before you trust an AI tool in your care

If you encounter an AI tool involved in your care, you can ask for four things in plain words. What does it do for me. What does it not do. Why did it suggest this for my situation. Who is responsible for the decision that follows. Any tool worth using should have someone able to answer all four without reaching for jargon, and a good care team will welcome the questions.

The goal of explainability is not to turn patients into engineers. It is to keep people informed, calm, and included in decisions about their own bodies.

This article is general education and not medical advice. For questions about your own health or any tool involved in your care, please talk with your own clinician, who knows your full situation.