AI is quietly becoming part of everyday medicine.

Inside the electronic health record systems that hospitals use, AI tools now flag patients at risk of sepsis, predict who might be readmitted, or suggest who needs attention first.

These tools can help. But they can also go wrong: making inaccurate predictions, performing worse for some groups of patients, or failing when used on populations they weren’t built for.

That’s why US rules now require developers to publicly explain how they manage those risks.

A new study checked whether they actually do.

The answer: not consistently. Yet every developer was certified anyway.

Developers must publicly report how their AI tools handle risk

Developers of electronic health record systems that include AI-powered “predictive decision support” tools must publicly report how they handle risk. That includes:

  • risk analysis: identifying what could go wrong
  • risk mitigation: what they do to reduce those risks
  • governance: who oversees the tools and how

They must also report on eight additional attributes, which under the federal rules cover qualities such as validity, fairness, safety and privacy.

Meeting these requirements is part of getting the software certified under the federal health IT program.

The reports from 35 developers were inconsistent and often incomplete

Researchers at Georgetown University and MedStar Health’s National Center for Human Factors in Healthcare reviewed the public disclosures of 35 electronic health record developers.

They found that reporting was “inconsistent and often incomplete.”

  • Some disclosures were detailed. Others were thin.
  • They varied widely in structure, making them hard to compare.
  • Many didn’t clearly explain specific risks or how they were managed.

And yet all 35 developers had been certified as meeting the requirements.

Compliance isn’t the same as clarity

The bigger lesson here is about the gap between ticking the box and giving people what they need.

A hospital choosing an AI tool needs to know practical things: Was it tested on patients like ours? Does it perform worse for certain groups? What happens when it’s wrong? Who is responsible for monitoring it?

Certification is supposed to give some assurance. But if developers can pass with very different levels of disclosure, then “certified” doesn’t tell a hospital much about how well risks are explained.

AI safety also depends on more than a model’s accuracy in testing. It depends on how risks are identified, monitored and communicated once the tool is used in the messy reality of a hospital.

What the study doesn’t show

This is a peer-reviewed study, published as a brief communication in npj Digital Medicine. The detailed results weren’t publicly available at the time of writing, so this article is based on the paper’s summary.

Thin reporting doesn’t mean bad practice. A developer may manage risk carefully behind the scenes and simply describe it poorly in public.

It measured transparency, not safety. The study didn’t test whether any AI tool was inaccurate, biased or caused harm.

It covered 35 developers in the US. Results may differ elsewhere or change as rules evolve.

Certification shouldn’t be read as proof that risks are known

The authors call for standardized reporting and more rigorous certification review.

For hospitals and health systems, a reasonable takeaway is not to treat certification as proof that an AI tool’s risks are well understood. It makes sense to ask developers for the evidence directly: how the tool was validated, on which patients, and how its performance is monitored over time.

Transparency rules may be falling short

As AI spreads through healthcare, transparency rules are one of the main ways hospitals and patients can understand what these tools do and where they might fail.

This study suggests those rules are only as strong as their enforcement.

A certificate can say a box was ticked without saying what’s in the box.