Corrections in New Zealand has moved quickly to reprimand staff after an internal breach of its AI rules, saying use of generative tools to draft formal, personal-information‑bearing reports is “unacceptable” — a development that spotlights how public-sector organisations are wrestling with the tension between productivity gains and privacy, accuracy and legal risk. ps://www.privacy.org.nz/news/contact-us/using-ai-to-contact-us/)
The incident, described in a recent RNZ report and confirmed by Corrections leadership, involved staff using Microsoft’s Copilot Chat on Department-issued devices to assist with formal casework — including Extended Supervision Order reports — contrary to the department’s own AI policy. Corrections says the department halted and investigated the activity, reminded staff of strict prohibitions against entering personal or health information into Copilot Chat, and emphasized that only the limited, tenant‑bound CoMicrosoft 365 is allowed on its network.
Corrections’ posture reflects a common public‑sector approach: allow a single, administratively manageable AI assistant under enterprise controls, block consumer or third‑party chat tools, and govern use through policy, training and auditing. That model has real advantages — but it depends on consistent staff behaviour, robust technical configuration and clear recordkeeping. Microsoft documents show that Microsoft 365 Copilot Chat offers enterprise data protection, logging and option controls designed for organisations, but these technical facilities do not remove the need for governance, human oversight or legal compliance.
Key technical points public organisations should note:
Microsoft’s own documentation warns that generative assistants can be inaccurate and that users should verify outputs. That guidance is necessary but not sufficient; organisations need processes that make verification mandatory for rights‑affecting or legally significant documents.
If Corrections wants to make AI adoption an enduring productivity gain rather than a recurring privacy and reputational headache, it will need to couple the current policy posture with enforceable technical configurations, a transparent remediation path for incidents, and sectoral standards that make human verification and provenance mandatory for every rights‑affecting document. The RNZ reporting has made the gap visible; the next step is institutionalising the controls that close that gap and restore public confidence.
Source: RNZ Corrections takes action against staff's 'unacceptable' use of artificial intelligence
Background / Overview
The incident, described in a recent RNZ report and confirmed by Corrections leadership, involved staff using Microsoft’s Copilot Chat on Department-issued devices to assist with formal casework — including Extended Supervision Order reports — contrary to the department’s own AI policy. Corrections says the department halted and investigated the activity, reminded staff of strict prohibitions against entering personal or health information into Copilot Chat, and emphasized that only the limited, tenant‑bound CoMicrosoft 365 is allowed on its network.Corrections’ posture reflects a common public‑sector approach: allow a single, administratively manageable AI assistant under enterprise controls, block consumer or third‑party chat tools, and govern use through policy, training and auditing. That model has real advantages — but it depends on consistent staff behaviour, robust technical configuration and clear recordkeeping. Microsoft documents show that Microsoft 365 Copilot Chat offers enterprise data protection, logging and option controls designed for organisations, but these technical facilities do not remove the need for governance, human oversight or legal compliance.
What Corrections did (and why it matters)
What the department says happened
- Corrections restricted AI on its network to Microsoft Copilot (Copilot Chat) and blocked other public AI applications.
- Copilot Chat was introduced on Corrections devices in November 2025; uptake since then is reportedly around 30% of staff.
- The department’s AI policy explicitly bans entering personal, identifying, health or case-management details into Copilot Chat, and prohibits using Copilot Chat to draft or generate content for reports or assessments containing personal information.
- Where breaches occurred, Corrections opened a privacy risk assessment, audited prompt logs, and issued internal reprimands and clarifications to staff and managers. Corrections also says it has an AI assurance of cybersecurity) and participates in the All‑of‑Government AI community and governance forums.
Technical and legal context
Microsoft Copilot: enterprise features and limits
Microsoft’s enterprise Copilot Chat provides enterprise data protection (EDP), conversation logging and tenant‑scoped controls that make it technically easier for organisations to manage prompts and responses. Copilot Chat logs prompts and can be configured to limit which organisational content the assistant can access; Microsoft also documents that Copilot Chat does not use enterprise prompts to train its base models. Those controls are helpful, but they are not a substitute for process, training and legal compliance.Key technical points public organisations should note:
- Prompts and responses can be logged and retained under Microsoft’s enterprise policies — that audit trail is valuable for compliance and incident review.
- Copilot Chat may use web grounding (search queries to Bing) to produce answers that reference the web; those web queries can be visible in the chat and are retained for a limited period. This introduces an external‑exposure vector.
- Enterprise data protection and regional data boundaries exist, but traffic may be routed to other regions under load; data‑residency commitments are contractual and have caveats for web‑grounded queries. Organisations must understand these details before assuming total data isolation.
Privacy law and the Office of the Privacy Commissioner (OPC)
New Zealand’s Privacy Act applies to the use of personal information, including when processed by AI systems. The OPC has warned that agencies must understand the technologies they adopt and ensure use meets privacy requirements. When an agency claims that its policy prohibits personal data entering an AI chat tool, the OPC’s focus will be on whether breaches occurred and whether the agency took appropriate remedial action. That is, policy alone is not enough — enforcement, auditing and notification procedures matter.Comparative examples: why public bodies worry about Copilot and similar tools
AI‑related incidents in other public-sector contexts underline the practical risks. In the UK, a policct included an erroneous factual claim that subsequent inquiries traced back to AI‑generated output from a Copilot‑style assistant; the mistake highlighted how an unverified AI output crept into operational briefings and then into governance decisions. That episode prompted immediate suspensions, audits and policy tightening — a playbook that mirrors Corrections’ reaction in New Zealand. The international example shows the shape of the failure mode: AI output → insufficient human verification → operational use → public trust and legal consequences.Microsoft’s own documentation warns that generative assistants can be inaccurate and that users should verify outputs. That guidance is necessary but not sufficient; organisations need processes that make verification mandatory for rights‑affecting or legally significant documents.
Strengths in Corrections’ approach
Corrections’ response contains a number of sound, defensible elements:- Constrained deployment: allowing only Microsoft Copilot Chat on the Corrections tenant is a sensible first‑line control because it reduces the risk surface compared to unmanaged consumer AI services.
- Explicit policy language: banning personal data in prompts and forbidding Copilot Chat to draft reports sets a clear behavioural standard for staff.
- Auditability: logging and the ability to search prompts gives Corrections the means to investigate incidents and enforce policies.
- Governance structures: appointing an AI assurance officer within cybersecurity and participating in the government‑level AI community shows organisational intent to integrate AI governance with broader public‑service guidance.
Gaps, risks and potential liabilities
Despite the strengths above, the incident exposes a set of ongoing and material risks:1) Human behaviour and policy enforcement
Policies are effective only if staff understand and ftions incidents show that even explicit bans can be ignored, whether through misunderstanding of what is allowed, the convenience of shortcuts, or pressure of caseloads. That creates risk of non‑compliance and privacy breach.2) Hallucination and factual integrity
Generative models produce plausible but sometimes false statements. In a corrections context — where assessments, risk reports and court‑facing documents influence liberty, conditions of supervision and rehabilitation decisions — a fabricated or misattributed claim can have severe legal and ethical consequences. Human verification is essential; reliance on AI drafts without rigorous checks is a systemic hazard.3) Privacy, data flows and third‑party processing
Even with enterprise data protection, sending prompts to a cloud service means third‑party processing. The legal status of that processing, the exact locations of LLM calls, and the handling of web search grounding must be contractually and operationally understood. If sensitive health or identity details are transmitted, the agency could face obligations under privacy law — and in some cases, required notification to the OPC or affected individuals. Corrections has reportedly not yet made a formal notification to the OPC; the decision whether to notify will depend on the scope and sensitivity of any data transmitted.4) Records management and evidence chain
When Copilot Chat or any AI assistant is used to generate or edit a report, organisations must be able to demonstrate provenance: who produced what, what AI assistance was used, what edits were made, and when. Public bodies are legally required to manage official records; failure to capture provenance risks misplacing responsibility and undermines evidentiary integrity in tribunals or courts.5) Overreliance and procurement creep
The initial, limited deployment of Copilot might tempt teams to expand use into higher‑risk areas (intelligence reports, risk assessments, legal briefings) without adequate procurement, red‑teaming or contractual safeguards. The organisational incentives — speed and efficiency — can drive this scale‑up unless it is tightly controlled.Practical recommendations for Corrections — and any public body
These recommendations are practical, sequenced, and tailored to the specific risks visible in this case.Immediate (0–30 days)
- Issue a temporary moratorium on using Copilot Chat for any content that includes personal information or that will be incorporated into formal reports or court documents. Make the moratorium explicit and time‑limited to allow rapid policy clarification.
- Audit and triage recent Copilot Chat logs to identify all prompts that included personal or health data. Use the logs to find affected files and interviews; determine whether remedial notices are required. (The auditability of Copilot Chat is an asset here; Microsoft documents that prompts and responses can be logged and retained under enterprise policies.)
- Notify legal and privacy teams and prepare a thresholded plan for OPC notification if the triage identifies breaches of personal information that meet notification criteria. Engage early with the OPC for guidance on remediation and reporting.
Short term (1–3 months)
- Revise and publish operational guidance that specifies permitted and forbidden Copilot uses with concrete examples (what is acceptable: grammar editing of non‑personal internal templates; what is not: drafting ESOs or assessments).
- Mandate a provenance field in every rights‑affecting document where staff must record whether and how AI was used, who reviewed the content, and which primary sources corroborate each factual claim.
- Run role‑based training and prompt‑scepticism exercises for community corrections teams, including mock audits that show how AI hallucinations can appear and how to catch them.
- Apply technical enforcement: block file‑upload features, disable web‑grounded queries for Copilot Chat where appropriate, and tune admin settings to restrict access to open‑content features that could allow Copilot to ingest internal case files inadvertently.
Medium to long term (3–12 months)
- Procure stronger contractual safeguards when licensing AI assistants: require auditable logs, data residency commitments, model‑behaviour warranties, and red‑teaming evidence from vendors.
- Embed recordkeeping automation so that any AI‑assisted drafting flow automatically produces metadata about prompts, sources and reviewer sign‑off that becomes part of the official record.
- Establish sectoral standards with the Government Chief Digital Officer’s All‑of‑Government community for how generative AI can be used in rights‑affecting public services; align to the Public Service AI Framework and National AI Strategy guidance. These cross‑agency standards should include mandatory human verification and provenance rules.
What success looks like: measurable controls
Public bodies can move beyond binary “ban or allow” debates by setting measurable success criteria for safe AI adoption:- 100% of rights‑affecting documents include an AI‑use provenance field.
- Zero instances of personal health or identifying information sent in AI prompts without explicit, pre‑approved mitigation.
- Quarterly AI‑use audits that are reported toe board and redacted summaries made available to oversight bodies.
- Mandatory training completion for all staff in roles that touch on case documentation, with annual refreshers and scenario‑based assessment.
Ethical and reputational considerations
Beyond compliance and technical risk, there is an ethical dimension. Corrections holds intensely sensitive information about individuals’ liberty, health and risk. Even a correctly logged and technically secure AI prompt can erode trust if briefing documents, assessments or court filings contain content that AI generated and ied. For Corrections, professional integrity of assessments is not just legal hygiene — it is central to how the community, courts and tangata whai i te raro (people under supervision) trust the system. Corrections’ emphasis on protecting privacy and professional integrity is therefore necessary, but the department must back that rhetoric with demonstrable, documented controls and visible remediation where breaches occur.Lessons for other organisations
The Corrections case is a practical case study for any organisation introducing generative AI tools:- Do not assume enterprise branding equals risk elimination. Vendor assurances and enterprise controls reduce, but do not remove, privacy and accuracy risk.
- Log and retain prompts: an auditable prompt history is a critical control for both privacy incident response and for understanding how AI outputs migrated into final documents.
- Treat AI output as work in progress, not authoritative fact. Organisations need mandatory human verification, especially where outcomes affect rights, safety or legal status.
- Align with national guidance. In New Zealand the GCDO anI Framework offer sectoral guidance that agencies should be integrating into practice.
Conclusion
Corrections’ swift reaction — tightening guidance, auditing prompt logs and reinforcing a prohibition against using Copilot Chat for personal‑information‑bearing reports — is the right first move. The broader lesson is that technology choices alone do not ensure safety. Technical protections from vendors like Microsoft are useful and often necessary, but they must be embedded in processes, training, procurement safeguards and records management that reflect the legal and ethical stakes of the work.If Corrections wants to make AI adoption an enduring productivity gain rather than a recurring privacy and reputational headache, it will need to couple the current policy posture with enforceable technical configurations, a transparent remediation path for incidents, and sectoral standards that make human verification and provenance mandatory for every rights‑affecting document. The RNZ reporting has made the gap visible; the next step is institutionalising the controls that close that gap and restore public confidence.
Source: RNZ Corrections takes action against staff's 'unacceptable' use of artificial intelligence
