Clinicians and AI: What They Actually Ask

Contents11 sections

Treatment comes before diagnosis

When clinicians use an AI tool at work, they mostly ask what to do. Diagnosis comes well behind. That's the finding of a study in JAMA, the Journal of the American Medical Association, published online on 15 October 2026: physicians asked about treatment and management in 33.1% of their classified queries, more than twice the 13.9% they spent on diagnosis.

Most public tests of medical AI ask it to make a diagnosis. So the tests and the real work point at different questions, and the gap widens with a look at who's asking. Many questions fell outside the asker's own specialty. More than 40% of users weren't physicians at all. The study is large. It's useful. It also comes from one company's platform, labelled by a language model, with three of its seven authors on that company's payroll.

What physicians asked

Patel and six colleagues analysed queries entered on OpenEvidence, an AI clinical information platform, between 1 January 2024 and 30 June 2026. Over those thirty months, 952,008 clinicians typed about 271 million queries. Of those clinicians, 556,200 were physicians, which is 58% of the users.

Just over half of physicians' queries (51.9%) concerned clinical decision-making. Treatment and management was the biggest category inside it. In the paper's words, "The most frequent query purposes were treatment and management (33.1%) and medication information (14.1%)." Diagnosis came third, at 13.9%. Questions of clinical knowledge took 37.2%, evidence 7.0%, administrative tasks 3.0% and medical education 0.9%.

The pattern makes sense once it's stated. A clinician usually knows the diagnosis, or suspects it. The harder daily question is what to do about it in this patient: which drug, what dose given the kidney function, what interacts with what's already prescribed, what the latest guideline says. That's where the reference gets opened — usually with the patient still in the room.

Outside their own field

Many questions were about topics outside the asker's specialty. The share ran from 35% for oncology and hematology up to 88% for emergency medicine. Surgeons aimed 70% of their queries at non-surgical topics, and anesthesiologists sent 73% outside anesthesiology.

That's what a quick reference is for. Specialists know their own field and look up the rest, which is sensible, and it also means the tool gets used most heavily in exactly the territory where its user is least able to notice when it's wrong.

Not only physicians

Non-physicians made up more than 40% of users and typed about 36.5% of the queries. The study counts 171,034 nurse practitioners, 93,241 physician assistants, 35,819 pharmacists and 95,714 other licensed clinicians. Nurse practitioners alone outnumber the pharmacists nearly five to one.

An evaluation that imagines a single physician user is testing for a little over half the real population.

How the study was built, and who built it

Nobody read 271 million queries. The authors drew a stratified random sample of about 38.8 million (a sample taken separately from defined groups, so each group is represented), which is roughly one query in seven. Then a large language model, rather than a person, labelled each one.

The authors listed their own limits. Read them.

• One platform. The data come from one tool, so they undercount clinicians' total AI use and may not generalise; only queries typed into the main interface were captured.
• Uneven denominators. Reliable workforce counts existed only for physicians, so other professions were compared against national provider registrations, which include people who no longer practise.
• Machine labels. A language model assigned the purpose and topic labels, so some queries are misclassified, and a generalist's question often fits several specialties.
• No context. Nobody could tell whether a query was asked during a visit, beforehand, or away from clinical work.
• No causal claim. The design can't say whether AI replaced other sources, changed what clinicians want to know, or stood in for a referral.
• Publisher content. The platform has agreements with major medical publishers, so its use may differ from tools without that content.

Then there's the authorship.

Three of the seven authors were OpenEvidence staff during the study: Samuel Finlayson as a salaried employee, Mondira Ray as a full-time employee, and Travis Zack as chief medical officer. The paper discloses all of it. The other four authors work at Harvard Medical School, Brigham and Women's Hospital, Massachusetts General Hospital and NYU, and none of those institutions ran the platform the data came from or held its logs.

Company authorship doesn't make the numbers wrong; access to a company's logs often requires it. It does mean nobody independent checked the data. That matters, because the company gains from a finding that its tool is widely used. A replication on another platform would settle it.

What a useful evaluation would test

Many published evaluations of medical AI score diagnostic accuracy, often on exam-style vignettes (short written cases with one right answer). On at least one widely used platform, real demand looks different, and an evaluation built for that demand would change in four ways.

• Weight treatment and drugs. Dosing, interactions and therapy choice should count at least as much as diagnosis.
• Ask outside the field. Questions should come from outside the asker's specialty, where an error is hardest to spot.
• Test the real mix of users. Nurse practitioners, physician assistants and pharmacists ask different questions in different words.
• Grade the answer someone would act on. The reply and its sources matter, beyond whether a label is right.

Two changes to MARA

MARA, the Medical Agentic Resident Assistant this project is designing, gathers a patient's record, checks it and recommends, and a person makes every decision. The study changed two parts of her plan. Her first functions now put treatment and medication questions ahead of diagnosis. Her design also names nurse practitioners, physician assistants and nurses as primary users, beside physicians.

MAX4, the desktop system MARA is being developed from, will weight its own test set the same way. Neither change proves anything. Both decide what gets tested first.

What nobody knows yet

• Other tools may differ. Nobody knows whether the same mix of questions holds on other tools, in other countries, or where AI is built into the health record itself.
• Accuracy wasn't measured. The study counts what was asked, and says nothing about whether the replies were right or what clinicians did with them.
• The labels are unchecked. How far the model's labels differ from what people would assign isn't reported.
• Outcomes weren't measured. Whether patients were better off is outside the design.

Why it matters

A model can top every diagnostic benchmark and still fail the nurse practitioner who wanted a renal dose at two in the morning. The questions clinicians actually type are about acting, often outside their own field, which is exactly where a wrong answer travels furthest before anyone catches it. Grading medical AI on the question people rarely ask tells a buyer very little about the one they ask every shift.

Sources

• Patel VR, Finlayson SG, Liu-Hass M, Ray M, Burk-Rafel J, Zack TI, Jena AB. "Clinician Use of an Artificial Intelligence–Enabled Clinical Information Platform." JAMA, published online 15 October 2026. doi:10.1001/jama.2026.21868. https://jamanetwork.com/journals/jama/fullarticle/2855491
• Healthcare AI Guy Weekly, 20 October 2026, which pointed to the study. https://www.healthcareaiguy.com/p/healthcare-ai-guy-weekly-10-20

Glossary