Recent reporting that ChatGPT and other large language models (LLMs) routinely invent or mis‑attribute sources is not clickbait: a peer‑reviewed study and an international audit both show troubling failure modes that should reshape how researchers, journalists, educators and everyday users treat AI-generated citations and news answers. The headlines — fabricated bibliographic entries, broken or misleading DOIs, and confidently wrong news summaries — are supported by systematic testing, and the practical takeaway is simple but urgent: always verify.
Background / Overview
A November 2025 news item on Zamin.uz summarized a new academic analysis alleging widespread citation fabrication by ChatGPT and warned that broadcasters’ audits find similar problems in news‑focused prompts. The Zamin piece reported that “35 (19.9%) of the 176 citations” produced in AI‑generated literature reviews were fabricated and that many of the remaining references contained errors such as incorrect page numbers, missing DOIs, or wrong publication dates. The article also cited a broadcaster‑led audit that found frequent factual and sourcing errors across assistants. Those numbers track a formal experimental study by researchers at Deakin University (published in JMIR Mental Health) that systematically prompted GPT‑4o to generate six short literature reviews in mental‑health topics, extracted 176 citations, and then verified each reference against academic databases. The Deakin/JMIR team found that 35 of the 176 citations (19.9%) were fabricated — meaning no identifiable publication could be located — and that a large share of the remaining citations contained bibliographic errors such as invalid or incorrect DOIs. The authors concluded that citation fabrication and bibliographic mistakes are common, especially for less‑prominent or highly specialized topics, and recommended mandatory human verification and stronger editorial safeguards when LLMs are used in scholarship. Independently, a coordinated audit led by the European Broadcasting Union (EBU) and run with public broadcasters found that AI assistants produced answers with material problems in nearly half of tested news queries: roughly 45% of assistant replies had at least one significant issue — errors serious enough to mislead readers — with sourcing failures and temporal staleness being the largest contributors. The audit pooled blind reviews from professional journalists across 22 public broadcasters in 18 countries and covered multiple assistants (including ChatGPT, Microsoft Copilot, Google Gemini and Perplexity). Taken together, these projects paint a clear picture: LLMs can be fast and fluent — but they are not reliable substitutes for primary sources or editorial fact‑checking.What the Deakin study actually did (and what it found)
Methodology in brief
The research team prompted GPT‑4o in June 2025 to generate six literature reviews (~1,100–1,300 words each) on three psychiatric conditions that vary in public familiarity and research maturity: major depressive disorder, binge eating disorder, and body dysmorphic disorder. Each condition was covered in a general review and in a specialized review focused on digital interventions. The model was instructed to provide bibliographic citations (≥20 citations per review). Researchers then attempted to verify all 176 citations using Google Scholar, Scopus, PubMed, WorldCat and publisher databases.Key verified results
- Total citations generated: 176.
- Fabricated (no identifiable source): 35 citations — 19.9% of the total.
- Among the 141 non‑fabricated citations, bibliographic errors were common: wrong DOIs, incorrect page ranges, misplaced publication dates and author‑list mistakes were frequent. The most common error type was DOI problems.
- The rate of fabrication varied by topic familiarity: major depressive disorder (well‑established research base) produced far fewer fabricated citations than the less visible disorders. The authors found higher fabrication for specialized prompts in less mature subfields.
How the EBU/BBC audit complements and broadens the concern
The Deakin study focuses on scholarly citations; the EBU/BBC audit targets news integrity. Both converge on a common theme: models produce fluent outputs that can be confidently wrong.- The EBU/BBC audit had journalists pose the same 30 news‑related questions to multiple assistants and blind‑review thousands of replies. Its core headline: about 45% of assistant answers contained at least one significant issue; around a third of replies exhibited serious sourcing failures (missing or misleading attribution); and roughly 20% contained major factual or temporal errors.
- The audit documented vivid real‑world failure modes: temporal staleness (outdated facts presented as current), mistaking satire for fact, misattribution of sources, and invented events or quotes. One such example reported in the audit was models continuing to present the wrong current office or public figure (a temporal error that journalists flagged as materially misleading).
Why these failures happen (technical anatomy)
Understanding the root causes makes mitigation practical rather than fatalistic.- Retrieval vs. generation mismatch: production assistants often combine a retrieval layer (search), a generative LLM, and a provenance layer. When retrieval returns low‑quality, partial or stale documents, the generator still synthesizes a confident answer, sometimes inventing citations to fill gaps. This misalignment is a technical root cause identified by auditors.
- Post‑hoc provenance assembly: some systems attach or reconstruct citations after composing an answer instead of composing strictly from retrieved, verifiable passages. That post‑hoc reconstruction can create ceremonial citations that look authoritative but don’t support the claim.
- Helpfulness bias: many models are tuned to minimize refusals and maximize helpfulness. That optimization increases the chance a model will answer rather than say “I don’t know,” producing plausible but ungrounded claims.
- Sparse training signal in niche domains: as Deakin’s study shows, the less the topic appears in the model’s training corpus, the higher the risk of fabricated or erroneous citations. Specialized prompts and narrow literatures create a vacuum the model fills with plausible‑sounding but incorrect fabrications.
Strengths of the evidence (what makes these results credible)
- Experimental rigor: the Deakin/JMIR study used explicit prompts, cleared chat history between reviews, and attempted exhaustive verification of each provided citation across major academic indexes — a transparent, reproducible approach.
- Editorial realism: the EBU/BBC audit used professional journalists across many countries and languages and applied newsroom standards (accuracy, sourcing, context, separation of opinion and fact), producing operationally meaningful measures for newsrooms and public information contexts.
- Cross‑validation in independent reporting: multiple reputable outlets summarized the same findings, reinforcing that these are not one‑off lab artifacts but systemic patterns observed under realistic prompts.
What this means for researchers, journal editors and educators
The implications are practical and immediate.- For researchers: treat any LLM‑produced citation as a lead, not evidence. Every reference the model supplies must be verified against primary databases (CrossRef, PubMed, Scopus, WorldCat). Recording prompts, model version and timestamps should become standard practice for reproducibility.
- For journal editors and peer reviewers: require authors to certify that any AI‑generated references were checked and to provide verification logs or DOI resolution proof where appropriate. Journals should explicit policy: AI may be used to draft text but not to invent scholarly citations.
- For educators: teach students that instant fluency ≠ authority. Assignments that allow AI use must demand primary‑source verification and transparent reporting of tools, prompts and checks.
Practical mitigations and a short playbook
- Mandatory human verification: every LLM‑sourced citation, DOI or web link must be checked by a human before being used in a paper, presentation, or public communication. This is non‑negotiable for scholarly and professional contexts.
- Retrieval‑first workflows (RAG): use retrieval‑augmented generation where the model composes answers from explicitly retrieved, timestamped documents that are attached to the response. RAG still needs human inspection, but it reduces fabrication risk by anchoring outputs to concrete sources.
- DOI/CrossRef checks: automated verification tools should resolve every DOI and check authorship and metadata against CrossRef or publisher APIs. If a DOI resolves to an unrelated article or fails, flag the citation as invalid. Deakin’s study shows many fake DOIs resolve to unrelated items — an automated DOI resolution step would catch many of these failures.
- Editorial provenance requirements: products embedded in newsrooms or academic workflows should expose the material the model used (machine‑readable provenance), and platforms should offer a “verified mode” that refuses to answer when provenance is weak. The EBU audit recommends similar provenance transparency for news tasks.
- Training and policies: organizations must adopt AI‑use policies that enforce verification steps, require documentation of AI interactions (prompts, timestamps, tool versions), and include AI‑literacy training focused on these failure modes.
Caveats and unverifiable or overstated claims
- Misnaming of the university: the Zamin piece refers to “Dikin University,” but the peer‑reviewed study was conducted by researchers at Deakin University (Australia) and published in JMIR Mental Health. That looks like a translation or transcription error in the reporting. Readers should consult the original study rather than secondary summaries when accuracy matters.
- “Up to 40% fabrication” phrasing: Zamin wrote that the EBU noted chatbots “fabricate up to 40%” of responses. The EBU/BBC audit’s headline was that ~45% of responses contained at least one significant issue and ~31–33% had sourcing failures; different rounding and simplifications across outlets explain the “up to 40%” language, but it’s an imprecise summary. Use the audit’s detailed metrics for operational decisions.
- Vendor‑specific dynamics: the audit and Deakin’s experiment were snapshots tied to specific model versions and test windows. Vendors regularly update models and retrieval pipelines, so absolute rankings change. However, the pattern — that fluency masks grounding failures — is robust across versions and vendors.
Deeper risks and systemic implications
- Erosion of research integrity: fabricated references erode the scaffolding of scholarly discourse. If AI‑sourced literature reviews are accepted without verification, false leads may propagate, wasting researcher time and polluting subsequent syntheses.
- Newsroom amplification: assistants used as quick news summarizers can compress hedging into categorical claims, misattribute quotes, or invent events. When audiences accept those summaries without reading original reporting, errors propagate at scale. The EBU audit demonstrates this risk in a practical newsroom setting.
- Legal and regulatory exposure: organizations that deploy assistants for external or semi‑external communication expose themselves to reputational and legal risk if AI outputs are used as factual bases (for policy, legal filings, or safety communications) without verification. The cautious stance adopted by several public broadcasters is instructive: treat assistant outputs as leads, not evidence.
A realistic outlook: where AI helps and where it must be constrained
AI assistants are powerful productivity tools when used as drafting aids, brainstorming partners, or for summarizing large corpora — with human oversight. They can accelerate literature scanning, generate useful first drafts, and surface candidates for deeper inspection. But they should not be the final authority on facts, citations, or breaking events.- Use AI for: exploratory scanning, drafting, ideation, translating tone, and preparing verifier‑friendly lists for human checking.
- Do not use AI for: final bibliographies, unverified citations in publications, sole sourcing of news summaries for public distribution, or legal/medical/financial decisioning without human expert sign‑off.
Recommendations for vendors, institutions and power users
- Vendors: expose retrieval evidence and timestamps; offer a “verified” response mode; and invest in stronger citation‑generation pipelines that validate DOIs and metadata against publisher APIs before returning them.
- Academic publishers: require AI‑use statements, verification logs, and confirmation that human experts validated every AI‑generated citation; consider automated DOI resolution gates in submission workflows.
- Newsrooms and public broadcasters: integrate editorial checks for any AI‑sourced reporting, insist on machine‑readable provenance, and prioritize refusal or conservative answers for time‑sensitive queries where provenance is weak.
- Individual researchers and students: verify every cited source against primary databases, log prompts and model versions, and retain AI interaction records for peer review and reproducibility.
Conclusion
The twin diagnostics — an experimental study showing nearly one in five citations fabricated in model‑generated literature reviews and a multinational newsroom audit showing roughly 45% of news replies with a significant issue — are a clear and convergent signal: LLMs are powerful writing tools but not reliable citation or news authorities. The risk is not cryptic; it’s practical and preventable with modest changes to workflows: require human verification, use retrieval‑anchored systems, validate DOIs and metadata automatically, and insist on provenance transparency.The Deakin study and the EBU/BBC audit do not mean AI is useless; they mean AI demands disciplined integration. Treat model outputs as provisional drafts and research leads, not as final, citable facts. That rule — verification first — should now be standard operating procedure in every research lab, newsroom, classroom and enterprise that uses conversational AI.
Source: Zamin.uz How reliable are ChatGPT data? New analysis response - Zamin.uz, 24.11.2025
