Artificial intelligence is now being used inside local children’s social care to transcribe and draft case notes — and practitioners are raising alarm after finding hallucinated content in machine-generated records that, in some cases, invents sensitive claims about children’s mental health and family situations. These errors are not marginal editing slips: social workers and independent researchers say they can change the factual record on which safeguarding decisions, assessments and long-term plans depend, and that the systems in active pilot across councils have produced phrases such as “suicidal ideation” or odd imagery (“fishfingers… flies… trees”) that were never spoken in an interview.
Local authorities in England and Scotland have been trialling audio-capture and generative-AI drafting tools to reduce administrative burden on overstretched social work teams. Vendors promise that speech-to-text and summarisation features allow frontline workers to be fully present during visits and produce near-complete case notes quickly afterwards, freeing time for supervision and direct work with families. Early pilot evaluations and vendor trials report measurable time savings and accessibility benefits for neurodivergent practitioners, who say AI support reduces the cognitive load of drafting dense, regulated records.
Those operational gains are why tools such as Magic Notes (operated by Beam) and enterprise assistants integrated with Microsoft Copilot have been adopted in dozens of local authority pilots and procurements. The stated model of use is that AI produces a first draft, which the social worker reviews, edits and signs off before the content becomes an official record. But research and practitioner testimony show that in practice this human verification step is not always enforced or sufficiently rigorous, which creates room for error to enter the official file.
Yet the downside is systemic when the wrong governance choices are made. Hallucinations are not cosmetic; when they alter facts or insert clinical claims, they can change the course of a child’s life. The risk increases when agencies treat AI outputs as authoritative or when workflows remove the time and responsibility necessary for human verification. This is not only a technical problem but an institutional one: procurement, contract law, professional standards and workplace practices must align to contain the hazard.
There is also an equity concern. Models trained on skewed corpora may over‑represent certain socioeconomic or cultural frames, producing assessments that systematically misinterpret language or behaviour in marginalized families. This risk demands transparency from vendors and rigorous bias testing by procuring authorities.
Local authorities should not be judged by the speed gains alone. They must ask: do we have the time, training and audit processes to ensure that every AI-generated line in a child’s record has been checked, contextualised and accepted by a qualified professional? If the answer is anything other than a confident “yes,” the sensible course is to pause and fix governance before moving from pilot to scale. The welfare and rights of children depend on records that reflect what was truly said — not what a model guessed.
Source: The Cool Down Social workers report 'hallucinations' found in AI transcriptions of accounts from children: 'All these words … have not been said'
Background: why councils turned to AI transcription and drafting tools
Local authorities in England and Scotland have been trialling audio-capture and generative-AI drafting tools to reduce administrative burden on overstretched social work teams. Vendors promise that speech-to-text and summarisation features allow frontline workers to be fully present during visits and produce near-complete case notes quickly afterwards, freeing time for supervision and direct work with families. Early pilot evaluations and vendor trials report measurable time savings and accessibility benefits for neurodivergent practitioners, who say AI support reduces the cognitive load of drafting dense, regulated records.Those operational gains are why tools such as Magic Notes (operated by Beam) and enterprise assistants integrated with Microsoft Copilot have been adopted in dozens of local authority pilots and procurements. The stated model of use is that AI produces a first draft, which the social worker reviews, edits and signs off before the content becomes an official record. But research and practitioner testimony show that in practice this human verification step is not always enforced or sufficiently rigorous, which creates room for error to enter the official file.
What researchers and social workers found: hallucinations, omissions and tone changes
Examples that matter
Independent study and reporting during recent eight‑month reviews documented multiple concrete failure modes:- Invented clinical statements — in at least one reported instance, an AI transcription added a reference to suicidal thoughts that the child had not expressed, a serious misrepresentation with immediate safeguarding implications.
- Nonsensical substitutions — when a child described parental conflict, some AI-generated notes substituted unrelated words like “fishfingers” or “flies,” producing a transcript that did not reflect the semantics of the conversation.
- Added content and tone-shift — social workers reported that when they asked the tool to redraft notes with a different tone, the assistant inserted “all these words that have not been said” — a direct quote used by a practitioner to describe the experience.
Independent verification: an eight‑month study
The Ada Lovelace Institute’s multi-month review — which evaluated live deployments and practitioner experience — concluded that “some potentially harmful misrepresentations of people's experiences are occurring in official care records.” The report emphasised that generative systems produce plausible-sounding text that can be factually incorrect or contextually inappropriate (the classic definition of a hallucination), and it called for clear governance and stronger auditing of systems used in social care settings.Why these hallucinations happen: the technical and workflow causes
Model behaviour and plausible-sounding error
Generative models — whether used to transcribe audio to text or to summarise and rephrase — optimise for plausibility and coherence, not guaranteed factual fidelity. When audio is noisy, a speaker mumbles, or there are interruptions, the speech-to-text layer can misrecognise words. The downstream language model then smooths and fills gaps, producing fluent sentences that may include invented details to make an internally consistent narrative. In effect, the model is guessing to resolve uncertainty — and those guesses can become institutionalised if not corrected.Training data and representational bias
Models reflect the data they were trained on. If vendor defaults or prompts bias the assistant toward particular phrasings or risk frames, the tool can inadvertently introduce disproportionate emphasis on clinical language (for example, mental-health indicators) or culturally specific interpretations. This is particularly hazardous in social work where language and context must be interpreted with care. The absence of transparency about training corpora and vendor fine-tuning compounds the problem, because procurement teams and practitioners cannot independently audit the patterns the model will favour.Workflow and governance gaps
The operational model — “AI drafts, human signs off” — is safe only when organisations enforce the human-in-the-loop step with adequate time, training and audit trails. Investigations of public-sector AI incidents (including other high‑profile Copilot errors in policing contexts) show common procedural lapses: lack of prompt-and-output logging, absence of mandatory verification steps, and insufficient training. Where staff are stressed and time-poor, initial drafts may be accepted casually or edited superficially, allowing hallucinated content to survive into the official record.The stakes: why AI inaccuracies are especially serious in children's social care
Legal, clinical and ethical harm
Records in children’s social care inform legal decisions about contact, placement, protection and, ultimately, the child’s future. A false note about self-harm or abuse can lead to unnecessary removal from the home, inappropriate referrals to mental-health services, or incorrect risk stratification that follows a child through subsequent assessments. Even allegedly small fabrications can have outsized consequences when they persist across a child’s file or are treated as corroborating evidence by other agencies.Trust, trauma and the therapeutic relationship
Social work relies on trust: children and families must be willing to share sensitive details. If families later discover that records contain invented phrases or mischaracterisations, that trust can collapse. For children who have experienced trauma, being misquoted or having false clinical language attached to their file can be re-traumatising and may affect therapeutic engagement long-term. These human costs are harder to quantify than time-savings metrics but are central to the ethical calculus.Amplification across systems
Public-sector records are shared across agencies: health, education, police, and courts. A hallucination in a social-care note can propagate into multi-agency meetings, become the trigger for a statutory assessment, or be cited in legal proceedings. The more integrated and automated these flows become, the higher the risk that an AI error will cascade into consequential downstream decisions.What vendors and employers say — and what that means in practice
Vendors: drafting tools, not decision-makers
Vendors commonly position their products as drafting assistants and advertise features such as hallucination checks or guarded defaults. For example, providers of note-taking tools emphasise that the system produces a first draft and that the practitioner is responsible for the final record. Some vendors claim to run automatic checks for hallucinations or to avoid using customer data to train public models. But product assurances alone do not prevent errors entering files if organisational controls are weak or if staff lack the time and training to apply rigorous verification.Employers and professional bodies: shared responsibility
The British Association of Social Workers (BASW) and other professional bodies are urging caution: practitioners must check records and retain professional judgment over content entering files, and employers must provide policies, training, and clarity about accountability. There have been reports of disciplinary action where AI-drafted notes were not properly verified before being stored as official records. Professional guidance emphasises that legal and ethical responsibility for a record cannot be outsourced to an algorithm.Governance, procurement and data-protection imperatives
Contractual safeguards procurement teams must insist on
- No unauthorised model training: explicit contractual prohibition on using client audio or transcripts to further train vendor or third-party models.
- Data residency and encryption: guarantees of in-region storage, encryption at rest and in transit, and clear retention policies.
- Auditability and prompt/output logging: durable, exportable logs of raw audio, transcription output, prompts sent to models, and model responses for every record that enters a child’s file.
- Independent testing: vendor-supplied models should undergo external bias and hallucination testing, with results auditable by the procuring authority.
Operational controls and professional safeguards
- Mandatory verification step: design workflows so a human must explicitly approve and sign off every AI-produced note before it is added to the official file.
- Time allocation: employers must protect protected time for verification — accepting AI should not be used to justify fewer minutes for clinical reflection and editing.
- Training and competency: staff must be trained to recognise hallucinations, check provenance, and dispute or correct model outputs.
- Safeguarding overrides: conservative default settings should flag any clinical language (e.g., suicide, self-harm, abuse) for mandatory corroboration before inclusion.
Privacy impact and regulatory duties
Under UK data-protection frameworks and professional confidentiality rules, public bodies must perform Data Protection Impact Assessments (DPIAs) when deploying new high-risk processing of sensitive categories (special category data, health information, etc.). DPIAs should explicitly address whether audio or transcript data will be used for model training, retention durations, and third-party access. Procurement must align contract terms with the DPIA conclusions and be auditable.Practical checklist for councils and social care providers
- Stop and map: catalogue where and how AI transcription/drafting is used, which teams and vendors are involved, and what data flows exist.
- Require DPIAs and independent audits for each tool before real-world scaling.
- Enforce signed human verification: no AI output becomes an official record without an audit trail showing who verified and why.
- Preserve raw audio: retain the original recording for a specified retention period so disputes can be resolved against primary data.
- Train staff on hallucination detection and record correction workflows; protect time for verification.
- Contractually prohibit vendor model training on client data and demand exportable logs and forensic access.
Broader technical remedies and design recommendations
Make models auditable and conservative by default
Systems used in social care must prioritise provenance and conservative defaults over fluency. That means:- Attach provenance metadata and confidence scores to every claim an AI makes.
- Where confidence is low, output an explicit “uncertain” marker or refuse to auto-complete sensitive phrases.
- Provide side-by-side access to the raw transcript and the AI-draft, with highlights for any inserted text that was not present in the audio.
Sandbox and adversarial testing
Before deployment, models should be stress-tested with adversarial prompts that probe hallucination tendencies and bias. Independent red-team reviews and public-sector FOI disclosures in other contexts show that pre-deployment testing and transparent reporting of failure modes materially reduces the risk of catastrophic errors reaching operational use.Human-centred UI design
Design interfaces so that users cannot accidentally accept an AI’s text without reviewing the audio and edits. Enforce friction where the system detects sensitive terms: require a checklist or free-text rationale for any acceptance of content flagged as clinical or risk-related.Analysis: balancing benefits, risks and professional integrity
AI tools offer real operational benefits — time savings, inclusion for neurodivergent practitioners, and the potential for more immediate, accurate record-keeping when used correctly. Multiple pilots and vendor reports show social workers reclaiming minutes previously spent on write-ups and reinvesting that time in supervision and direct work. These productivity and wellbeing gains are meaningful for an overstretched workforce.Yet the downside is systemic when the wrong governance choices are made. Hallucinations are not cosmetic; when they alter facts or insert clinical claims, they can change the course of a child’s life. The risk increases when agencies treat AI outputs as authoritative or when workflows remove the time and responsibility necessary for human verification. This is not only a technical problem but an institutional one: procurement, contract law, professional standards and workplace practices must align to contain the hazard.
There is also an equity concern. Models trained on skewed corpora may over‑represent certain socioeconomic or cultural frames, producing assessments that systematically misinterpret language or behaviour in marginalized families. This risk demands transparency from vendors and rigorous bias testing by procuring authorities.
What regulators and professional bodies should do next
- Issue clear, prescriptive guidance on acceptable procurement and operational practices for AI transcription in social care, including mandatory DPIAs and audit requirements.
- Require exportable forensic logs and retention of raw audio to enable dispute resolution and audits.
- Define red lines: kinds of decisions or records that must never be automated or autogenerated without multi-party human corroboration (for example, any record that asserts suicidal ideation, serious abuse, or a statutory threshold being met).
- Support training and accreditation so that social workers remain the professional authors of case reasoning and assessment.
Conclusion: proceed, but only under strict human-centred guardrails
AI transcription and drafting tools can make social work more efficient and inclusive — but the technology is not neutral or infallible. When a machine-generated sentence can invent a child’s suicidal thought or change the meaning of a disclosure, policymakers, procurement teams and practitioners must treat the tool as an assistive drafting technology only, with robust human verification, contractual safeguards and clear professional accountability.Local authorities should not be judged by the speed gains alone. They must ask: do we have the time, training and audit processes to ensure that every AI-generated line in a child’s record has been checked, contextualised and accepted by a qualified professional? If the answer is anything other than a confident “yes,” the sensible course is to pause and fix governance before moving from pilot to scale. The welfare and rights of children depend on records that reflect what was truly said — not what a model guessed.
Source: The Cool Down Social workers report 'hallucinations' found in AI transcriptions of accounts from children: 'All these words … have not been said'
