Artificial-intelligence “slop” is now producing real-world waste: archivists, librarians and researchers are spending growing amounts of time chasing non‑existent papers, journal series and archival records that large language models invent on demand — and the problem is escalating into legal, scholarly and operational headaches for institutions that must prove a negative.
Generative AI models — from OpenAI’s ChatGPT to Google’s Gemini and Microsoft’s Copilot — are trained to produce fluent, plausible text. That fluency makes them powerful assistants for drafting, brainstorming and summarizing, but it also hides a crucial limitation: they do not perform primary‑source verification and will often fabricate details that look authoritative, including bibliographic citations, archive call numbers and entire journal titles. The International Committee of the Red Cross (ICRC) publicly warned researchers that chatbots “may generate incorrect or fabricated archival references,” noting the models’ propensity to invent catalogue numbers, document descriptions, and even platforms that never existed. The phenomenon has immediate, measurable consequences. Reporters found that state and university archivists are receiving an increasing share of reference requests that were generated by AI — requests that sometimes point to fabricated journals or unique primary‑source documents. One reported estimate cites roughly 15 percent of emailed reference questions to a state archive as originating from ChatGPT, including hallucinated citations that are difficult for staff to disprove. That figure, and the broader trend, are forcing institutions to change workflows, introduce verification requirements and limit the amount of staff time they will spend chasing unverifiable claims.
Source: Scientific American AI Slop Is Spurring Record Requests for Imaginary Journals
Background / Overview
Generative AI models — from OpenAI’s ChatGPT to Google’s Gemini and Microsoft’s Copilot — are trained to produce fluent, plausible text. That fluency makes them powerful assistants for drafting, brainstorming and summarizing, but it also hides a crucial limitation: they do not perform primary‑source verification and will often fabricate details that look authoritative, including bibliographic citations, archive call numbers and entire journal titles. The International Committee of the Red Cross (ICRC) publicly warned researchers that chatbots “may generate incorrect or fabricated archival references,” noting the models’ propensity to invent catalogue numbers, document descriptions, and even platforms that never existed. The phenomenon has immediate, measurable consequences. Reporters found that state and university archivists are receiving an increasing share of reference requests that were generated by AI — requests that sometimes point to fabricated journals or unique primary‑source documents. One reported estimate cites roughly 15 percent of emailed reference questions to a state archive as originating from ChatGPT, including hallucinated citations that are difficult for staff to disprove. That figure, and the broader trend, are forcing institutions to change workflows, introduce verification requirements and limit the amount of staff time they will spend chasing unverifiable claims. Why AI Makes Up References: A Plain‑English Technical Primer
The generative objective and the “always‑answer” problem
Large language models are statistical sequence predictors optimized to continue text in a way that is likely given their training data. They are not search engines: unless explicitly connected to a reliable retrieval layer, they will fabricate plausible continuations when the correct factual material is absent. Because the objective is to produce a fluent response rather than to assert only when certain, the models often generate false but plausible bibliographic entries and archival descriptors. This is a structural limitation, not merely a tuning error.Hallucinations vs. errors
- Hallucinations: Confident assertions with no factual basis (e.g., a non‑existent journal name or a fabricated archive call number).
- Errors: Mistaken facts that reflect imperfect knowledge of real entities (e.g., the wrong publication year for a real paper).
Retrieval‑Augmented Generation (RAG) helps — when implemented properly
A reliable approach is RAG: the model consults a curated database or live index and conditions its output on retrieved documents. Properly engineered RAG pipelines can dramatically cut hallucination rates for citations by forcing the model to ground its answers in verifiable records. However, not all consumer chatbots use robust RAG, and even when they do, retrieval quality and index freshness vary — meaning hallucinations remain a live risk.How the Problem Is Manifesting: Archives, Libraries and Courts
Archives and special collections: the burden of proving non‑existence
Archivists face a unique burden. A fabricated citation to a “unique primary source” requires an archivist to search finding aids, accession registers and sometimes decades of uncatalogued material to show that an item does not exist. The ICRC specifically called out AI‑generated archival references as a growing issue, and state and university archives are reporting rising volumes of AI‑originated reference questions that consume staff time. Institutions are now asking researchers to disclose AI use when making requests and asking them to pre‑verify sources.Academic integrity: students and scholars
Universities confront a twin threat: students who rely on AI-generated references uncritically can submit papers riddled with fabricated citations, and faculty who use LLM assistance may inadvertently include bad references in literature reviews. The reproducibility and traceability of scholarship depend on verifiable references; fabricated citations can mislead peer reviewers and amplify false claims into the academic record.Courts and legal practice: sanctions and reputational risk
The legal profession has already seen aggressive consequences when AI hallucinations migrate into official filings. Recent reporting documents dozens of cases where attorneys submitted briefs citing non‑existent cases generated by chatbots; judges in multiple jurisdictions have publicly admonished counsel, fined firms and warned that reliance on unverified AI research can amount to malpractice. The legal domain shows how fabricated citations are not a mere nuisance but can carry financial sanctions and ethical consequences.Evidence: What the Data and Research Say
- Controlled evaluations across multiple chatbots found that fewer than 30 percent of generated academic references were entirely correct, with a large share partially correct or wholly fabricated. This empirical result underlines that hallucinated citations are a model‑class property, not an isolated bug.
- Natural‑language research communities have focused on reference hallucinations as a measurable research problem. Work presented at major conferences demonstrates diagnostic tests and probing strategies to detect when a model is likely fabricating references — a promising research frontier for automated detection.
- Institutional and journalistic accounts demonstrate real operational effects: public archives and reference desks report higher workloads and new policies; courts have penalized litigants for submitting AI‑generated false citations; and faculty and students have been embarrassed and corrected after reliance on AI‑produced bibliographies.
Strengths and Opportunities: What AI Does Well in Research Workflows
AI tools remain valuable when used correctly. Key strengths include:- Rapid synthesis: Models condense large literatures into readable summaries and identify potential search terms and themes.
- Drafting and editing: For well‑documented areas, LLMs accelerate drafting, paraphrasing and language polishing.
- Discovery suggestions: Even when their citations are imperfect, models can surface relevant keywords, author names and topics that guide productive human search.
Risks and Failure Modes
- False authority: Hallucinated citations are typically fluent and confidently presented, increasing the risk that non‑expert users will trust them.
- Resource drain: Archivists and reference librarians spend scarce time chasing nonexistent items, diverting resources from genuine research assistance.
- Scholarship contamination: Fabricated citations that slip into theses, articles or briefs can misdirect follow‑on work and amplify false leads.
- Legal/ethical exposure: Professionals who submit unverified AI‑generated references can face sanctions, reputational damage and professional discipline.
- Scaling misinformation: Because models can invent many plausible but false items quickly, the scale of the problem can outpace manual verification if institutions do not adapt.
Practical Guidance: How Institutions and Researchers Should Respond
For researchers, students and independent writers
- Treat AI as a brainstorming tool, not an authority. Use chatbots to generate search terms, topical overviews and draft language — but not to cite primary sources without verification.
- Verify every reference. Before including a citation in any formal work, confirm it via:
- Library catalogues and discovery layers
- Publisher databases, CrossRef and DOI lookups
- Primary archive finding aids or direct contact with archivists
- Disclose AI use in research workflows. Where institutions permit AI assistance, state what was AI‑generated and what was human‑verified.
- Keep an audit trail. Document searches and verification steps for key references so you can substantiate claims if challenged.
For librarians, archivists and information professionals
- Update intake workflows. Require requesters to indicate if an AI tool produced a citation and to provide the raw prompt or output when possible.
- Set realistic service limits. Communicate clear boundaries for staff time spent verifying unverifiable claims.
- Educate users. Provide training materials and short guides on AI hallucinations and how to verify sources quickly.
- Leverage technology. Deploy or pilot RAG systems for internal reference tools so staff can triage likely valid leads faster.
For universities, publishers and professional bodies
- Adopt verification policies. Mandate that any AI‑aided references must be human‑verified prior to submission or publication.
- Institute penalties for negligent reliance. Mirror legal sector guidance by clarifying professional responsibility for accuracy, even when using AI tools.
- Support development of verification tools. Fund or pilot automated citation‑verification research and services that cross‑check references against trusted bibliographic databases.
Technical and Product Remedies: What Developers and Vendors Should Do
- Expose provenance. Models should clearly label when an output is based on retrieved documents vs. model imagination. Provenance metadata must be native to the reply.
- Switch to conservative modes for references. When asked for citations, models should either:
- Retrieve and display exact metadata with clickable identifiers (DOI, handle, catalogue number), or
- Explicitly state uncertainty and refuse to invent a citation when no reliable result is found.
- RAG by default for factual claims. Integrate up‑to‑date, curated indexes for archives, journals and legal reports and make RAG the default behavior for reference queries.
- Built‑in verification APIs. Offer developer APIs that let downstream services verify citations against CrossRef, WorldCat, national archive catalogues and legal databases before presenting them to users.
Case Studies: Real Incidents and Institutional Reactions
- The ICRC published a public advisory after archivists reported repeated encounters with fabricated document references produced by mainstream chatbots. The advisory emphasized checking official online catalogues and established scholarly references rather than trusting AI output. The ICRC also warned about the increasing difficulty of convincing users that a requested unique record does not exist.
- State archives and libraries report a measurable uptick in AI‑generated queries. One widely reported example cites a state library estimating that roughly 15 percent of emailed reference questions are ChatGPT‑generated; the library is implementing policies to require source vetting and disclosure. This operational shift illustrates how user behavior is changing faster than institutional processes. Note: the 15 percent figure is reported in journalistic coverage of library staff accounts; direct institutional data releases are uneven, so the precise percentage may vary by location and over time.
- In multiple U.S. court cases, attorneys submitted filings containing non‑existent case law generated by AI; judges have fined attorneys and admonished counsel for relying on unverified AI output. These incidents have catalyzed bar associations to reiterate that lawyers remain responsible for the accuracy of their filings regardless of tools used.
A Checklist for Safer Use of AI in Research and Archival Work
- Always ask: "Can I find this citation in a trusted catalogue or database right now?"
- If the AI provides a DOI, CrossRef lookup or stable URL, verify it immediately.
- For archives: ask the user to provide the AI prompt and the model output; require that staff spend no more than a pre‑announced time budget searching for unverifiable records.
- For teaching: require students to submit verification screenshots or links for every source generated with AI assistance.
- For legal work: make independent confirmation from primary legal databases mandatory before citing any case or statute suggested by an AI tool.
What We Still Don’t Know — and What to Watch
- Precise prevalence by sector remains uncertain. Different studies and institutional reports show widely varying rates of hallucinated citations depending on prompts, model versions and retrieval setups; local experience may differ from aggregate studies.
- Vendor progress on grounding and provenance is uneven. Keep watching for product changes — especially new retrieval features, more conservative citation behavior, and explicit provenance metadata.
- Automated verification tools are promising but not yet ubiquitous. Research prototypes show good results, but broad deployment in editorial workflows, legal firms and libraries will take time and funding.
Conclusion
The rise of generative AI has delivered undeniable productivity gains, but the same fluency that makes these models useful also enables them to invent plausible‑sounding fabrications that can waste staff time, undermine scholarship and expose professionals to legal and ethical risk. The ICRC’s public advisory and multiple institutional reports show that this is not a hypothetical threat: fabricated archival and bibliographic references are already imposing costs. Addressing the problem requires action on three fronts: user education and verification discipline; institutional policy and workflow redesign; and vendor engineering to ground outputs and expose provenance. Until those three changes are broadly adopted, the safest rule for researchers and professionals remains simple and immutable: never accept an AI citation at face value — verify it before you rely on it.Source: Scientific American AI Slop Is Spurring Record Requests for Imaginary Journals
