Reporting Standards for Medical AI: What Makes a Claim Checkable

A medical AI claim is only as credible as it is checkable, and checkability comes from structured, complete reporting: what the tool is for, what it learned from, how well it performed and against what comparison, and where it is known to fail. State those four plainly and a reader can decide whether to believe the claim. Leave any of them out and the reader is left to take it on faith. Good reporting is the difference between "trust us" and "here, check."

I want to make the case for the spirit of good reporting rather than the letter of any one checklist. Several reporting frameworks now exist for clinical prediction models and AI studies, and they are useful. But they sit downstream of an older idea: a scientific claim should travel with enough information that a competent stranger can interrogate it without phoning you.

What does good reporting for medical AI actually mean?

A short definition I find useful: a medical AI claim is checkable when a reader can answer, from the report alone, who the tool is for, what it was built on, how it was measured, and when it should not be trusted. If a reader has to email the company to answer any of those, the report is incomplete. This is not about volume. A hundred pages that never state the intended population are worse than two pages that do. The point is that the load-bearing facts are present, findable, and honest enough that someone who wanted the tool to fail could use your own document to test it fairly.

Intended use: the claim everything else is measured against

The first thing a report must fix is intended use. What clinical question does the tool answer, for which patients, in which setting, used by whom, and feeding into what decision? Until that is settled, every performance number is unanchored, because you cannot call a tool accurate without saying accurate at what, for whom.

Intended use is also what makes a limitation meaningful. A model that reads chest images for adults is not flawed because it performs poorly on children; it was never offered for children. That protection only holds if the report said so in advance. A common failure mode is the quiet expansion of scope, where a tool validated for one narrow task gets described later as if it handled the whole disease. The intended-use statement is the fence that keeps a modest, real result from being inflated into a broad, unproven one. It is the first sentence I want written for any product, because it disciplines everything after it.

Training data: a model is an argument about its data

The second pillar is the data the model learned from, because a model is a compressed argument about the population it was trained on. If you do not describe that population, you have not described the model.

A useful report covers where the data came from, who is represented and who is not, how outcomes were defined and labeled, and how the data was split so that the test set was genuinely unseen. Each has a well-known way of going wrong. Draw all your data from a few large academic centers and the model may encode the habits of those centers rather than the disease. Let the test set leak into training and your number measures memorization rather than generalization. None of these traps implies bad faith; they are easy to fall into because they stay invisible in the final metric. The reader is not entitled to your raw records, but they are entitled to enough description to judge whether your data resembles their patients.

Performance: a number with a comparison and an interval

The third pillar is performance, and the central discipline is that a figure means little on its own. It needs a comparison and an honest expression of uncertainty. Compared to what? A tool can look strong against a weak baseline and ordinary against the standard of care a clinic already uses, so a report should say which it chose. Measured how? Accuracy alone can hide a model that does well on common cases and badly on the rare, dangerous ones, which is why sensitivity, specificity, calibration, and behavior across subgroups matter more than a single score. Across whom? A number from a site the model has never seen tells you far more than one from the kind of site it trained on.

Uncertainty deserves its own line. A point estimate with no interval invites the reader to treat a noisy result as a precise one. Reporting a confidence interval is not hedging; it tells the truth about how much the data can support. In the EASY-1 randomized controlled trial (NCT03258268) we ran for EASY Diabetes, the point was never a lone accuracy figure. It was that the tool was compared against standard of care, in a multi-clinic design, with prespecified outcomes, so the result could be read as an effect rather than an anecdote.

Limitations: the section that earns the rest

The fourth pillar is the one teams are most tempted to soften, and the one that does the most to make a report trustworthy. A clear statement of limitations tells the reader where the tool is known to be weak, which populations are thin in the data, and what kinds of error are most likely and most costly.

Stating limits feels like undercutting your own product. It is the opposite. A report that names its weaknesses shows the authors went looking for them, which is what makes the strengths believable. A report with no limitations is not a stronger claim; it is a less examined one.

Why structured reporting is the substance, not the paperwork

Report the four pillars with enough specificity and a reader can check you. Leave gaps and you are asking for trust you have not yet earned. This is why I treat reporting as part of the science, not as compliance overhead. The frameworks and checklists converge on one instinct, write the claim so it can be interrogated.

For clinicians, the practical move is to read the four pillars first and the headline number last. For builders, the most useful early question doubles as a reporting plan: what are we claiming, and what would let a skeptic check it. Answer that honestly and a credible report mostly writes itself.

This article is educational and reflects my own view as a researcher. It is not medical advice or regulatory guidance for any specific product, and clinical decisions should always involve your own clinician.