How to Judge Whether an AI That Writes Clinical Notes Is Any Good

An AI that drafts your clinic notes is only as good as the errors it hides, not the minutes it saves. Judge it on three things the marketing rarely mentions: whether it omits clinically important facts, whether it invents details that were never said, and whether it quietly changes the certainty of your language. Time saved is easy to measure and easy to oversell. Note accuracy is harder to measure, and it is the metric that decides whether a document is safe.

Ambient documentation tools listen to a clinical encounter and produce a draft note. The pitch is intuitive: less typing, more eye contact, shorter days. Because that pitch is about workflow, the evidence people cite is usually about workflow too. That is the first trap. A tool can save real time and still produce a document that is worse to read, harder to trust, or subtly misleading. So the honest question is not "does it save time" but "what does it get wrong, how often, and how badly."

Start with the outcome that is easy to game

Documentation time is the metric vendors reach for, and its appeal is easy to see. It is objective, it is captured automatically in the electronic record, and it moves in the direction everyone wants.

A pragmatic randomized trial run at UCLA and published in NEJM AI in late 2025 shows how uneven even this friendly metric can be. Investigators randomized 238 outpatient physicians across 14 specialties to two commercial ambient scribes or usual care, and measured time spent in the note using an Epic Signal metric. One tool cut time-in-note by roughly 9.5 percent versus control; the other showed no statistically significant reduction at all. Same category of product, same trial, opposite headline. If a single number can split like that between two vendors, a single number is not a verdict.

The design detail matters as much as the result. This was randomized, so the comparison is against a concurrent control group rather than against how a clinician felt last quarter. Most claims you will see are not randomized. They are before-and-after satisfaction surveys, which reliably flatter a new tool because novelty and self-selection both push in the same direction.

The metrics that actually protect the note

To judge accuracy you have to name the failure modes. Three matter most.

Omission is when the note leaves out something that was said and clinically relevant. It is the most dangerous error because it is invisible on the page. Nothing looks wrong; the fact is simply gone. Hallucination, sometimes called confabulation, is the opposite: the note asserts something that never happened, a symptom denied, an exam not performed, a value never spoken. Hedging drift is the quiet one. The patient said a symptom was "a little" better; the note records "improved." A finding described as "possible" becomes "present." The clinical certainty of the language shifts, and with it the meaning.

A 2025 framework published in npj Digital Medicine put numbers on the first two. Across nearly 13,000 clinician-annotated sentences from AI-drafted clinical text, the authors measured a hallucination rate of about 1.47 percent and an omission rate of about 3.45 percent. Omissions were roughly twice as common as fabrications, which fits the intuition that leaving things out is the more natural failure of a summarizing system. More important than the raw rates was the severity grading: around 44 percent of hallucinations were judged clinically major, meaning they could plausibly change diagnosis or management. A low error rate is not reassuring on its own. You need to know how many of those errors were the kind that could hurt someone.

That same UCLA trial catalogued the qualitative texture of these errors from physician reports: omissions, pronoun-resolution mistakes, failures to detect negation and affirmation, speaker misattribution, and structural clutter. Negation failure deserves a closer look. If a patient denies chest pain and the note records chest pain, the error is not a typo; it inverts the clinical picture.

Note bloat and the readability tax

Length is its own failure mode. Ambient tools tend to be generous, and a generous note is not a better note. When every utterance becomes a documented finding, the signal that a reader needs is buried under padding. A scoping review of ambient documentation metrics, posted to medRxiv in early 2025, found that most evaluation frameworks measure surface similarity to a reference text using natural-language metrics such as ROUGE and BERTScore, while few capture whether the note is clinically useful or appropriately concise. A note can score well on textual overlap and still be bloated, hedged, and tiring to read. What you want measured is whether a clinician can find the important facts quickly, not whether the draft resembles a template.

How to read a claim without being fooled

Bring a short checklist to any evidence a vendor or study offers.

Ask what was measured. If the only outcome is time or satisfaction, accuracy has not been tested. Ask about the comparison. Randomized against a control beats before-and-after every time, because the latter cannot separate the tool from novelty. Ask about omissions specifically, since they are the errors that leave no trace on the page. Ask whether errors were graded for severity, because a raw error rate without a harm scale tells you volume, not risk. Ask who read the notes, and whether clinicians blinded to the tool judged them, rather than the people who built or bought it. And notice what the scoping review made unavoidable: the field still lacks a shared standard for measuring the things that matter most, which means many published numbers are not comparable to one another.

None of this argues against ambient documentation. The technology can genuinely return time and attention to the encounter, and the npj framework showed that careful prompt and workflow design can push major-error rates below what is reported for unaided human notes. The point is narrower. A tool that drafts a medical record is a clinical instrument, and it deserves the scrutiny we give any instrument: not "did it feel faster" but "what does it get wrong, how often, and how much does that error cost."

This article is educational and is not medical advice.