Dental X-Ray AI Study: Free Copilot and Gemini Give Outdated Shielding Advice

What the study tested​

The researchers submitted 20 expert-reviewed English-language questions, using one standardized prompt, to a ChatGPT-5.5 configuration with Thinking enabled, the free Gemini 3 Flash configuration and the free Microsoft Copilot 2026 configuration, in late April 2026. That produced 60 responses, one per platform per question.

Other design details, from the paper and the Bioengineer report:

• Prompt: each system was told to answer as it would to a patient, in clear, simple language.
• Controls: every question got a fresh chat, with history and memory disabled. The same device, browser and network were used throughout.
• No retries: there were no follow-up prompts and no repeat generations. The study captured what each tool said once, not how consistent it is.
• Question selection: the 20 questions were narrowed from a pool of 57 after review by seven volunteer oral and maxillofacial radiologists, who did not later score the answers.
• Topics: radiation exposure, how often imaging is needed, risks, children, pregnancy, cone-beam CT, protection, myths, and when to consult a professional.
• Raters: three oral and maxillofacial radiologists scored coded responses with platform names removed.
• Status of the questions: the authors call the set expert-reviewed, not formally validated. They did not keep the item-level classifications or the content-validity calculations.

How scoring worked​

Accuracy, relevance, clarity, completeness, safety and triage appropriateness were scored from 1 to 5, and higher was better. Misinformation ran from 0 to 4, and lower was better. Keep that direction in mind when reading the misinformation results.

The safety domain covered the consequences of relying on an answer. Examples were false reassurance, unnecessary alarm, avoiding imaging that was indicated, and delaying professional care.

The statistics treated the 20 questions as the units of analysis. The three raters' scores for the same fixed answer are correlated, so they don't count as independent observations. This avoided treating the three correlated evaluator ratings as independent.

The headline results​

The ChatGPT configuration received more favourable ratings than Gemini and Copilot for accuracy, relevance, safety and misinformation. It also received more favourable clarity and completeness ratings than Copilot.

The figures reported for the three tools:

Other question-level results:

• Relevance: ChatGPT beat Gemini on 18 questions and Copilot on 19.
• Completeness: ChatGPT beat Copilot on all 20 questions. Against Gemini it tied on 17 of 20.
• Clarity: ChatGPT beat Copilot on 16 questions. Against Gemini it won 10, lost seven and tied three.
• Gemini vs. Copilot: Gemini rated better on clarity (10 questions) and completeness (19 questions). The two did not differ significantly on accuracy, relevance, safety or misinformation.
• Triage: only one question, about what a worried patient should do when a dentist recommends an X-ray, qualified. All three tools had a median score of 4. The paper declines to draw broader conclusions from that single item. (The citation index here is retained as transported; the figures come from the study's Results section.)

The gap between ChatGPT and Gemini on clarity and completeness was not statistically significant. A clean ranking across every dimension is not supported.

Why the ceiling effect matters​

ChatGPT scored the maximum on accuracy and safety in all 60 ratings. A scale that tops out this early cannot separate the best outputs from each other. The authors also say that many ties and few discordant questions produce extreme effect estimates. In plain terms, a very large effect size here mostly means the rank order was the same on nearly every question.

The authors add that perfect scores do not prove the text was free of outdated statements. Their own post hoc review found one, as the next section shows.

The lead apron problem​

After scoring, the authors did a separate descriptive review of the 60 responses for shielding advice. It did not change the original scores and was not blinded. They report:

• Outdated or unqualified advice about lead aprons or thyroid collars appeared in one ChatGPT answer, 12 Gemini answers and 11 Copilot answers.
• Claims of complete or near-complete protection appeared in six Gemini answers and one Copilot answer.
• Two examples were Gemini suggesting two lead aprons in pregnancy and Copilot saying aprons and collars are "always used."

That advice conflicts with current professional guidance. The American Dental Association said in February 2024 that dropping this was no longer a routine recommendation. More precisely, the ADA said lead abdominal aprons and thyroid collars are no longer recommended for patients during dental X-rays, regardless of age or health status such as pregnancy. It said modern digital equipment and limiting the beam to the area being imaged protect patients better. It also said shielding can block the primary beam and force retakes. The ADA tells dentists to follow all applicable federal, state and local regulations, so the rules where you live may still apply. The study cites the American Academy of Oral and Maxillofacial Radiology as aligned with this guidance.

A patient who has read "always wear a lead apron" may question a clinic that doesn't use one. The authors call that a plausible implication, not something they measured.

A scoring wrinkle​

The paper also notes an inconsistency in the raters' own scores. All 28 Gemini and 19 Copilot ratings with a misinformation score of 2 also received accuracy and safety scores of 4. The accuracy rubric reserves 4 for minor imprecision. The authors say this points to limits in how the rubric was applied. They kept the original scores unchanged and did not ask raters to explain them.

Why Copilot's showing needs context​

Copilot trailed in several domains, and this is not the first such result from overlapping authors. An earlier paediatric dental advice study by overlapping authors had ChatGPT highest on accuracy (M = 4.97), Gemini next (M = 4.60) and Copilot last (M = 3.97). A separate 2025 study in Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology tested ChatGPT-3.5, Gemini 1.5 Pro and Copilot on oral radiology questions. Professional websites and ChatGPT-3.5 did not differ on ease of understanding, and both scored significantly better than Copilot.

The pattern is consistent, but this study cannot say why. The configurations were unequal:

• ChatGPT used a paid tier with reasoning enabled. Gemini and Copilot used free tiers.
• Differences could come from access tier, how much reasoning the system spent, interface behaviour, retrieval features or undisclosed backend routing.
• The authors say plainly that the results do not show one underlying model family is better.
• They also did not keep exact timestamps, screenshots, exported conversations, browser builds or backend snapshots. Consumer AI products change often, so these exact results would be hard to reproduce.

A fair reading is that a reasoning-enabled paid ChatGPT answer beat free-tier answers from Gemini and Copilot on this narrow task. It is not a verdict on Copilot as a product, and it says nothing about Copilot in Windows, Microsoft 365 or paid tiers.

Other limitations​

• Question set: the questions were short, informational and English-language, and fairly homogeneous. Real patient questions are messier.
• Blinding: platform names were removed, but headings and style might have revealed the tool. Nobody tested whether the masking worked.
• Agreement: pooled intraclass correlation ranged from 0.696 for clarity to 0.976 for accuracy. Much of that agreement reflects large differences between platforms, not fine discrimination among answers within one platform.
• Clarity: radiologists judged it. No patients were tested, and no objective readability measure was used.
• Outcomes: the study did not measure comprehension, trust, anxiety, willingness to accept imaging, care-seeking or clinical outcomes.
• Publication status: the Bioengineer report is a summary of the journal paper. The journal's own page lists a different, longer title and says the final formatted version is still to come, so wording may change.

What this means for Copilot and other AI users​

Treat the findings as a caution about how you use these tools, not a ranking to cite.

1. Check anything involving radiation, pregnancy or children. Confirm it with your dentist or an authoritative source such as your national dental association.
2. Be sceptical of absolute language. "Always" and "completely protected" were the red flags here.
3. Compare like with like. A free-tier result and a paid reasoning-mode result are different products.
4. Ask your dentist why an X-ray is needed. The study's one triage question points the same way: ask about clinical justification instead of accepting or refusing on a chatbot's say-so.
5. Don't read "no misinformation" as "no errors." Even the top-rated configuration had one flagged shielding answer.

Bottom line​

On 20 patient-style dental X-ray questions, radiologists preferred a paid, reasoning-enabled ChatGPT configuration over free Gemini and free Copilot. The authors present it as a snapshot of single outputs under unequal conditions, not a durable ranking or clinical validation. The practical lesson is the shielding errors. Several free-tier answers repeated advice that dental bodies have moved away from, and readers should verify such claims with a professional.