A large-language-model study published August 5 in npj Health Systems found that an LLM assessing de-identified pediatric emergency-department notes showed a persistent tendency to rank younger children as more urgent than older children, even when the older patient had higher acuity. The failure was sharpest in infant-versus-adolescent comparisons: when the adolescent was the sicker patient, the model selected the adolescent only about half the time; when the infant was sicker, accuracy approached 90%.
That result is more consequential than a middling aggregate accuracy figure. A triage system’s job is to decide who should be seen first under scarcity, and an age-linked error in that decision can systematically delay care for a particular group. The study does not show that an LLM is ready to run a pediatric waiting room. It shows that an otherwise plausible use case—ranking patients by urgency—can conceal a repeatable skew that becomes most pronounced exactly where age differences are largest.
The researchers evaluated one LLM using pairwise comparisons: it received information from two pediatric ED visits and had to identify which patient was more acute. That framing is sensible for testing prioritization. Emergency departments do not need a software system to merely attach a number to a chart; they need to decide whether Patient A needs attention before Patient B.
But pairwise ranking also exposes a pattern that ordinary score-agreement metrics can blur. The model did not simply miss difficult cases at random. It more often elevated the younger child, and the tendency appeared to grow as the age gap between the two children widened. In practical terms, the model could be right about a genuinely ill infant while still being unsafe as a queue manager if it too readily pushes an adolescent with a serious condition lower in the line.
The paper’s most important contribution is therefore the age-stratified analysis, not the broad claim that the model achieved moderate accuracy. A system that is evenly wrong may be improvable with better calibration; a system that is predictably wrong in one direction creates an inequitable workflow by design. Hospitals would need to measure that behavior before deployment, then continually audit it after any model, prompt, data-feed, or workflow change.
This is also a warning for IT teams evaluating generative-AI pilots through a single dashboard number. Overall accuracy, AUC, and agreement with a prior score may look acceptable while a subgroup analysis reveals that the tool produces systematically different recommendations for infants, school-age children, and teenagers. In triage, that is not a secondary fairness metric. It is part of the core safety case.
That finding is independently consistent with a 2024 JAMA Pediatrics analysis of more than one million pediatric ED visits across 21 hospitals. The study found correct ESI v4 triage in 34.1% of visits, over-triage in 58.5%, and under-triage in 7.4%; under-triage was more likely for children age 6 and older than for younger children. Older adolescents had the highest relative risk of under-triage in that analysis.
The new LLM paper makes a defensible argument that this reference-standard limitation does not erase its finding. If the human score already leans toward treating younger children as more urgent, and the model still demonstrates an additional preference for younger children relative to that score, then the model’s age effect may be conservative rather than exaggerated. But the claim should not be stretched further than the evidence supports: the study does not establish a precise “true” rate of pediatric misprioritization, because its benchmark is itself shaped by clinician behavior and local triage practice.
This matters for vendors selling models as “aligned with clinician judgment.” Alignment can mean reproduction, not correction. In many clinical AI evaluations, matching historical decisions is treated as validation. Here, the historical decision process contains a known age-related problem. A model that accurately imitates it may pass a retrospective benchmark while preserving the very error pattern the hospital hoped automation would reduce.
For hospital technology leaders, the requirement follows directly: validation must include outcomes beyond historical labels. Admission, ICU admission, mortality, urgent interventions, escalation after arrival, and time to clinician assessment all provide different signals. None is a perfect definition of acuity, but relying only on past ESI assignments effectively tells the model that legacy triage practice is the ground truth.
The new study reports lower performance in pediatrics despite using newer models and prompting methods. That gap is the report’s most direct challenge to the common assumption that a stronger general-purpose model will automatically generalize to a smaller clinical population. Children are not simply lower-weight adults with different vital-sign ranges. Their physiologic norms change rapidly with age, their symptom reporting differs by developmental stage, and the clinical meaning of a fever, respiratory rate, or behavior can vary substantially between an infant and a teenager.
The authors suggest one likely explanation: broad LLM training data may contain less pediatric-specific information and fewer pediatric clinical examples than adult material. That is plausible, but it remains a hypothesis rather than a demonstrated causal mechanism in this study. The paper tested one model with one prompt, not its underlying training corpus or the effect of pediatric fine-tuning.
Still, the operational takeaway is clear. A successful adult benchmark is not a pediatric validation. Any health system considering clinical use of an LLM needs separate testing by age group and setting, rather than treating an adult emergency-department result as a transferable performance certificate.
That is especially true where an implementation turns language-model output into a rank, alert, recommendation, or order of work. The model may not assign a formal ESI level, yet if its output causes a nurse or clinician to re-prioritize a chart, it has affected a time-sensitive care decision.
The study’s own design, however, does not validate that use case. It is retrospective and draws from ED provider notes written later in the patient encounter than the initial triage assessment. The researchers attempted to approximate triage information by extracting selected note sections, but the time gap is fundamental. Later notes can include clinical observations, tests, responses to treatment, or evolution of symptoms unavailable at the front door.
The distinction is important. A system that can compare two static, retrospectively assembled records is not necessarily a system that can safely process streaming vital signs, missing data, delayed documentation, duplicated chart text, and rapidly changing patient status. In fact, continuous monitoring creates a new class of engineering and human-factors problems: alert thresholds, refresh intervals, queue churn, stale context, audit logs, downtime behavior, and clear accountability when the model and clinician disagree.
The paper appropriately calls for prospective, multicenter studies and regulatory evaluation before deploying LLMs in place of people at triage. It should go further in the minds of prospective buyers: the first prospective deployment should test whether the tool improves detection of deterioration without increasing false alarms, unnecessary room moves, or delays for older children. A model that preferentially escalates younger children could look “safer” in a narrow missed-critical-case metric while shifting attention away from adolescents who need it.
The FDA’s clinical decision-support framework also makes product intent relevant. Software that directs a clinician toward a specific patient-level action or is intended to replace clinical judgment can move beyond the boundaries of low-risk, non-device clinical decision support. A vendor cannot avoid that issue merely by describing the output as a “priority signal” if the intended workflow is to change who receives time-sensitive clinical attention.
The immediate response should not be to search for a prompt that tells the model to “ignore age bias.” Prompt wording can alter outputs, but it is not a substitute for a complete validation program. A deployable pediatric triage tool would need locked model and prompt versions, a defined input schema, subgroup performance thresholds, prospective shadow-mode testing, clinician override mechanisms, drift monitoring, incident review, and evidence that it improves care rather than only reproducing past documentation patterns.
The study’s evidence points toward a narrow conclusion: LLMs may eventually provide an additional safety signal in pediatric ED operations, but their ranking behavior cannot be assumed to be neutral simply because the input is de-identified clinical text. The infant-versus-adolescent result puts a concrete safety constraint on any such project. Before an LLM is allowed to influence the waiting-room queue, it must prove that a teenager does not lose priority merely because the patient beside them is younger.
The model learned a direction, not merely random error
The researchers evaluated one LLM using pairwise comparisons: it received information from two pediatric ED visits and had to identify which patient was more acute. That framing is sensible for testing prioritization. Emergency departments do not need a software system to merely attach a number to a chart; they need to decide whether Patient A needs attention before Patient B.But pairwise ranking also exposes a pattern that ordinary score-agreement metrics can blur. The model did not simply miss difficult cases at random. It more often elevated the younger child, and the tendency appeared to grow as the age gap between the two children widened. In practical terms, the model could be right about a genuinely ill infant while still being unsafe as a queue manager if it too readily pushes an adolescent with a serious condition lower in the line.
The paper’s most important contribution is therefore the age-stratified analysis, not the broad claim that the model achieved moderate accuracy. A system that is evenly wrong may be improvable with better calibration; a system that is predictably wrong in one direction creates an inequitable workflow by design. Hospitals would need to measure that behavior before deployment, then continually audit it after any model, prompt, data-feed, or workflow change.
This is also a warning for IT teams evaluating generative-AI pilots through a single dashboard number. Overall accuracy, AUC, and agreement with a prior score may look acceptable while a subgroup analysis reveals that the tool produces systematically different recommendations for infants, school-age children, and teenagers. In triage, that is not a secondary fairness metric. It is part of the core safety case.
The reference standard may understate the age effect
The study compared the LLM’s judgments with clinician-assigned Emergency Severity Index, or ESI, acuity levels. ESI is widely used in U.S. emergency departments, but it is a triage framework and an imperfect proxy for a child’s underlying urgency. The paper acknowledges a key complication: clinician-assigned ESI itself has documented age-related variation, with younger children more often over-triaged and older children more often under-triaged.That finding is independently consistent with a 2024 JAMA Pediatrics analysis of more than one million pediatric ED visits across 21 hospitals. The study found correct ESI v4 triage in 34.1% of visits, over-triage in 58.5%, and under-triage in 7.4%; under-triage was more likely for children age 6 and older than for younger children. Older adolescents had the highest relative risk of under-triage in that analysis.
The new LLM paper makes a defensible argument that this reference-standard limitation does not erase its finding. If the human score already leans toward treating younger children as more urgent, and the model still demonstrates an additional preference for younger children relative to that score, then the model’s age effect may be conservative rather than exaggerated. But the claim should not be stretched further than the evidence supports: the study does not establish a precise “true” rate of pediatric misprioritization, because its benchmark is itself shaped by clinician behavior and local triage practice.
This matters for vendors selling models as “aligned with clinician judgment.” Alignment can mean reproduction, not correction. In many clinical AI evaluations, matching historical decisions is treated as validation. Here, the historical decision process contains a known age-related problem. A model that accurately imitates it may pass a retrospective benchmark while preserving the very error pattern the hospital hoped automation would reduce.
For hospital technology leaders, the requirement follows directly: validation must include outcomes beyond historical labels. Admission, ICU admission, mortality, urgent interventions, escalation after arrival, and time to clinician assessment all provide different signals. None is a perfect definition of acuity, but relying only on past ESI assignments effectively tells the model that legacy triage practice is the ground truth.
Adult triage results did not transfer to children
The pediatric study was designed as a complement to a 2024 JAMA Network Open study by Christopher Williams and colleagues at UCSF. In the adult work, GPT-4 was tested on 10,000 pairs of ED presentations and correctly identified the higher-acuity patient in 89% of comparisons. On a 500-visit subset, its performance was reported as statistically comparable with participating emergency physicians.The new study reports lower performance in pediatrics despite using newer models and prompting methods. That gap is the report’s most direct challenge to the common assumption that a stronger general-purpose model will automatically generalize to a smaller clinical population. Children are not simply lower-weight adults with different vital-sign ranges. Their physiologic norms change rapidly with age, their symptom reporting differs by developmental stage, and the clinical meaning of a fever, respiratory rate, or behavior can vary substantially between an infant and a teenager.
The authors suggest one likely explanation: broad LLM training data may contain less pediatric-specific information and fewer pediatric clinical examples than adult material. That is plausible, but it remains a hypothesis rather than a demonstrated causal mechanism in this study. The paper tested one model with one prompt, not its underlying training corpus or the effect of pediatric fine-tuning.
Still, the operational takeaway is clear. A successful adult benchmark is not a pediatric validation. Any health system considering clinical use of an LLM needs separate testing by age group and setting, rather than treating an adult emergency-department result as a transferable performance certificate.
That is especially true where an implementation turns language-model output into a rank, alert, recommendation, or order of work. The model may not assign a formal ESI level, yet if its output causes a nurse or clinician to re-prioritize a chart, it has affected a time-sensitive care decision.
“Continuous reassessment” is a proposed use case, not a demonstrated capability
The authors argue that the safest initial role for an LLM could be continuous assessment: monitoring a waiting-room population and flagging patients whose priority should change as notes or vital signs evolve. As a product concept, that is more credible than replacing an experienced triage nurse with a one-time chatbot decision. A second signal that detects deterioration could be useful in a crowded ED.The study’s own design, however, does not validate that use case. It is retrospective and draws from ED provider notes written later in the patient encounter than the initial triage assessment. The researchers attempted to approximate triage information by extracting selected note sections, but the time gap is fundamental. Later notes can include clinical observations, tests, responses to treatment, or evolution of symptoms unavailable at the front door.
The distinction is important. A system that can compare two static, retrospectively assembled records is not necessarily a system that can safely process streaming vital signs, missing data, delayed documentation, duplicated chart text, and rapidly changing patient status. In fact, continuous monitoring creates a new class of engineering and human-factors problems: alert thresholds, refresh intervals, queue churn, stale context, audit logs, downtime behavior, and clear accountability when the model and clinician disagree.
The paper appropriately calls for prospective, multicenter studies and regulatory evaluation before deploying LLMs in place of people at triage. It should go further in the minds of prospective buyers: the first prospective deployment should test whether the tool improves detection of deterioration without increasing false alarms, unnecessary room moves, or delays for older children. A model that preferentially escalates younger children could look “safer” in a narrow missed-critical-case metric while shifting attention away from adolescents who need it.
The FDA’s clinical decision-support framework also makes product intent relevant. Software that directs a clinician toward a specific patient-level action or is intended to replace clinical judgment can move beyond the boundaries of low-risk, non-device clinical decision support. A vendor cannot avoid that issue merely by describing the output as a “priority signal” if the intended workflow is to change who receives time-sensitive clinical attention.
The deployment bar is higher than a better prompt
The study has other limitations that should shape how its results are used. It evaluated a single LLM and a specific prompt; different models, prompts, retrieval systems, structured inputs, and pediatric datasets may produce different behavior. Less than 1% of the evaluation material also overlapped with notes used for model selection and prompt optimization, which the authors say was unlikely to materially affect the overall result but remains a methodological detail future studies should eliminate.The immediate response should not be to search for a prompt that tells the model to “ignore age bias.” Prompt wording can alter outputs, but it is not a substitute for a complete validation program. A deployable pediatric triage tool would need locked model and prompt versions, a defined input schema, subgroup performance thresholds, prospective shadow-mode testing, clinician override mechanisms, drift monitoring, incident review, and evidence that it improves care rather than only reproducing past documentation patterns.
The study’s evidence points toward a narrow conclusion: LLMs may eventually provide an additional safety signal in pediatric ED operations, but their ranking behavior cannot be assumed to be neutral simply because the input is de-identified clinical text. The infant-versus-adolescent result puts a concrete safety constraint on any such project. Before an LLM is allowed to influence the waiting-room queue, it must prove that a teenager does not lose priority merely because the patient beside them is younger.
References
- Primary source: Nature
Published: 2026-08-05T00:00:00+00:00
Assessing acuity in pediatric emergency department triage: performance of a large language model | npj Health Systems
Pediatric triage performance varies across emergency departments (ED), contributing to ongoing challenges in pediatric emergency care. There is growing interest in using large language models (LLMs) to support more consistent triage decision-making in children. We evaluated an LLM’s (GPT-5-mini)...
www.nature.com
- Related coverage: publications.aap.org
Pediatric Triage Accuracy in Pediatric and General Emergency Departments | Hospital Pediatrics | American Academy of Pediatrics
BACKGROUND AND OBJECTIVES. Accurate triage at the time of emergency department (ED) presentation is critical for timely acuity assessment and anticipating resource requirements. Commonly, triage is conducted using the Emergency Severity Index (ESI); however, the accuracy of this approach for...
publications.aap.org
- Related coverage: divisionofresearch.kaiserpermanente.org
Current emergency room methods often overtriage children - Kaiser Permanente Division of Research
Kaiser Permanente analysis suggests more accurate triage approach could increase efficiency, shorten ER visits for low-acuity patients.divisionofresearch.kaiserpermanente.org - Related coverage: researchgate.net
(PDF) MAP: evaluation and multi-agent enhancement of large language models for inpatient pathways
PDF | Inpatient pathways require complex clinical decision-making based on comprehensive patient information, yet research on medical LLMs is limited in... | Find, read and cite all the research you need on ResearchGate
www.researchgate.net
- Related coverage: knowledge.uchicago.edu