Much of the debate about AI in healthcare rests on two opposite fears.
One is that patients distrust anything an algorithm decides. Psychologists call it algorithm aversion. The other is that people will only trust AI if it explains itself, the idea behind explainable AI.
Both ideas come mostly from laboratory experiments, where volunteers make hypothetical choices. A large new study tested them in the real world, with real patients and a real decision: whether to get a flu shot.
Telling patients they were at high risk got more of them vaccinated. Telling them a computer algorithm had made that call changed nothing at all.
Three flu seasons of field trials covered 90 000 patients
The trials ran at Geisinger, a non-profit health system in Pennsylvania. The research team, led by first author Gail Rosenbaum with senior authors Christopher Chabris and Michelle Meyer, published the results in Nature Human Behaviour.
A machine-learning model built by the company Medial EarlySign, trained on ten years of Geisinger patient records, scored each patient’s risk of catching flu and developing complications. It drew on more than 2 300 details from each patient’s records, such as lab results, vital signs and medical history. Its accuracy was good but far from perfect (an AUC of 0.79, where 0.5 is a coin flip and 1.0 is flawless).
Patients near the top of that ranking were enrolled in three preregistered randomized trials:
- 2020–21: the top 10% of risk, about 38 600 patients analysed
- 2021–22: the top 10% again, about 42 500 patients
- 2022–23: the next tier down, the top 11–20%, about 39 100 patients
Patients were randomly assigned to receive letters, patient-portal messages and texts, depending on their contact preferences. The key versions were:
- No study messages (the health system’s usual flu reminders only)
- A plain reminder to get vaccinated, with no risk information (added from the second season)
- “You’re at high risk”, with no explanation of how that was decided
- “High risk, based on a review of your medical records”
- “High risk, based on a computer algorithm-based analysis of your medical records”
In the second season, some messages also listed one or three personal reasons for the risk rating, such as “medical condition(s) related to breathing” or “medicine(s)”.
The outcome was simple and objective: did the patient get a flu shot, according to their medical records?
‘You’re high-risk’ messages raised vaccination by a few points
Risk messages worked. Compared with patients who got no study messages, those told they were high-risk were 1.7 to 3.5 percentage points more likely to get vaccinated, depending on the season. That’s a relative increase of roughly 3% to 15%. In the second season, for example, 34.1% of the high-risk group got vaccinated versus 31.5% of the control group.
Risk added something beyond a reminder. Against a plain reminder, the high-risk message added 1.1 to 1.4 percentage points. The authors estimate that about half the effect came from the risk information and half from simply being reminded. (In the second season this gap fell just short of the usual threshold for statistical significance in the main analysis, though it passed in preregistered robustness checks. In the third season it was clearly significant.)
A simple label was enough. In the first season, telling patients they were “in the top 10% of risk” worked no better than simply saying they were “at high risk”. A plain, qualitative warning did the job.
The algorithm made no difference. Across all three seasons, messages that named a “computer algorithm” worked about as well as messages that mentioned a records review or gave no source at all. In the first two seasons, the statistics pointed strongly toward there being no real difference, not just a failure to detect one. In the third season, the algorithm message did slightly worse than the plain high-risk message, but the difference wasn’t significant and the evidence was inconclusive either way.
Explanations made no difference either. Listing one or three personal reasons for the risk rating didn’t boost vaccination. The researchers dropped the reasons for the third season. They suggest that in similar settings, developers may get more value from making the algorithm more accurate than from making it explain itself.
The sweet spot was in the middle. Exploratory analyses suggested the risk message helped most for people in roughly the top 4–15% of risk. The very highest-risk patients may already know they’re vulnerable, perhaps because a doctor has told them. Those further down may not have believed it. The authors suggest health systems might send risk messages to the top 15%, with the very top 3% optional.
Letters beat texts. A fourth, smaller trial reported in the paper’s supplement compared ways of delivering the message. Printed letters, the most expensive option, were also the most effective, especially for older patients and those at the very highest risk.
Patients focused on the message, not the method
Why didn’t patients care where the risk estimate came from?
The researchers offer a few possibilities. People may simply focus on the part of the message that’s about them, “you’re at high risk”, and skim past how that was worked out. In a busy real-world inbox, the algorithm may also have been less noticeable than in a lab experiment, where it’s the whole point. In an online follow-up test, about three in four people correctly recalled where the risk rating came from, so most patients probably noticed.
It’s also possible that judging flu risk feels like an objective, number-crunching task, the kind people are comfortable handing to a computer.
Either way, the real-world result fits neither popular story. Patients weren’t scared off by the algorithm, and they weren’t impressed by it.
Put the effect sizes side by side and the point is clear. Famous lab studies of algorithm appreciation and aversion reported effects ranging from small to very large. Here, the differences between algorithm and non-algorithm messages were essentially zero. Meanwhile, the risk message itself performed on par with other successful nudges: a meta-analysis found letters raise target behaviors by about 2.4 percentage points on average, and text messages in a large flu-shot “megastudy” by about 2.1 points.
What the study doesn’t show
- Small effects. A few percentage points per message is typical for nudges, and worthwhile across a whole health system, but it won’t transform vaccination rates. The authors note that only a limited number of patients are “nudgeable”: unvaccinated, without a firm habit either way, and open to the vaccine.
- One health system. Geisinger’s patients are older, more rural and less ethnically diverse than many populations. Results may differ elsewhere.
- Incomplete records. Vaccinations given elsewhere may not all show up in Geisinger’s data, so some messages may have shifted where people got vaccinated rather than whether they did.
- Pandemic years. All three trials ran during COVID-19, which may have shaped attitudes to vaccination and to health messages.
- Wording matters. The researchers chose “computer algorithm” because it tested better than “artificial intelligence” or “machine learning” in a pre-survey. Other wording might land differently.
- A low-stakes decision. Flu shots are recommended for nearly everyone, so an algorithm getting it wrong costs little. People may react differently when an AI’s call has serious consequences.
- Attitudes to AI are changing fast. Patients in 2020–23 may not react the same way today.
- Commercial involvement. Geisinger paid Medial EarlySign to generate the risk scores and reasons used in the trials, which the paper discloses. The authors report no other competing interests.
The message mattered more than its source
Health systems increasingly use algorithms to decide who needs extra attention. This study suggests practical lessons for how they talk to patients about it.
First, tell people when they’re at high risk. A personal message works better than a generic reminder, and a simple “you’re at high risk” is enough. Second, being open about using an algorithm doesn’t seem to cost anything, at least for routine, low-stakes advice like a flu shot. Third, explanations may matter less than accuracy.
Patients asked whether the message was about them, and few seemed to care who, or what, had written it.