Verantos and Bayer's joint abstract featured in a Spotlight Session at ISPE's 42nd Annual Meeting in Milan

Large language models accurately inferred functional status after ischemic stroke from clinical notes, even after explicit modified Rankin Scale scores were removed. The models exceeded a prespecified 0.80 threshold for precision, recall, and F1 score at both discharge and follow-up. The work, a collaboration between Verantos and Bayer, is presented at ISPE 2026, the annual meeting of the International Society for Pharmacoepidemiology.

‍

Why it matters

The modified Rankin Scale is the standard outcome measure in acute stroke trials, but clinicians seldom record it during routine care. Manual chart review can recover scores from electronic health records, but it does not scale. Automated inference from clinical notes could make large real-world studies of stroke outcomes feasible.

How the models were tested

The study used electronic health record data from a single integrated delivery network in the US Midwest from 2018 through 2024. Eligible patients were adults with ischemic stroke documented during an inpatient or emergency department encounter who survived hospitalization.

• Discharge status came from physical and occupational therapy notes during the stroke admission. Follow-up status came from outpatient notes 7 to 425 days after the stroke.
• Notes containing explicit scores were selected, and two clinicians independently annotated each one to create a reference standard, with Cohen’s kappa of at least 0.70 required.
• Stated scores and their descriptions were redacted before model input so the models had to infer status from the surrounding narrative.
• Two open-source models, Mistral Small (22B) and Llama 3.3 70B Instruct, ran in a secure, privacy-preserving environment.
• Outputs were classified as independent (scores 0 to 2) or dependent (scores 3 to 5). Prompts were developed on training sets, tuned on validation sets, and evaluated on held-out test sets, with results averaged across three runs.

Key findings

• For discharge notes, the models achieved 0.84 precision, 0.83 recall, and an F1 score of 0.84.
• For follow-up notes, the models achieved 0.93 precision, 0.85 recall, and an F1 score of 0.88.
• At discharge, early models overestimated disability. Instructions to score conservatively when presentations were ambiguous corrected the bias.
• At follow-up, conservative prompting lowered recall instead. Physicians’ scoring showed consistent anchors, such as walker use or assisted-living residence indicating a score of 3, and adding these as prompt examples substantially improved performance.

What it means

Large language models can recover a rarely documented outcome measure from routine clinical notes with high accuracy. Performance differed by note type, so prompt strategy should reflect the documentation context. Because the study used score-enriched sampling, modest test sets, and data from a single health system, validation in unselected notes and additional health systems is the next step.