DBT's Microsoft 365 Copilot Pilot: High Satisfaction, Task Savings, No Dept Productivity Gains

The UK Department for Business and Trade’s pilot of Microsoft 365 Copilot returned a clear, measured verdict: staff liked the assistant, and specific writing and summarisation tasks became noticeably faster, but the trial produced no robust evidence that those time savings translated into department‑level productivity gains. (assets.publishing.service.gov.uk)

Background / Overview​

Microsoft 365 Copilot is an integrated generative‑AI assistant embedded into Word, Outlook, Teams, Excel, PowerPoint and a standalone Copilot Chat app. The Department for Business and Trade (DBT) ran a three‑month departmental pilot from October to December 2024, allocating 1,000 licences to UK‑based staff; roughly 70% of licences were taken by volunteers and the remaining 30% were randomly assigned to improve representativeness. The evaluation combined telemetry from Microsoft dashboards, a diary study, observed timed tasks and qualitative interviews to measure use, user satisfaction, and time savings. (assets.publishing.service.gov.uk)
A parallel, larger Government Digital Service (GDS) cross‑government experiment ran over a similar period, involving roughly 20,000 civil servants across 12 organisations and reporting different headline numbers. These two official pieces of work sit together in the public record and explain why media coverage has produced contrasting narratives about Copilot’s real‑world value. (gov.uk)

---

What DBT measured and how it measured it​

Mixed‑method evaluation design​

The DBT evaluation used three primary data streams:

• Telemetry from Microsoft’s M365 Copilot dashboard to capture active users and application use patterns.
• A diary study (three Excel sheets completed during a week in November 2024) collecting task‑level records of satisfaction, accuracy and estimated time without Copilot; the diary study achieved a 32% response rate. (assets.publishing.service.gov.uk)
• Observed tasks and interviews with a smaller sample to validate diary self‑reports and inspect output quality and verification overhead.

The evaluation adjusted time‑saving calculations to remove outputs that were unused and to subtract “novel” work (tasks performed only because Copilot made them possible), improving the realism of reported savings. Statistical checks were applied to test representativeness and subgroup differences. (assets.publishing.service.gov.uk)

Why method choice matters​

Self‑reported diaries and surveys tend to produce larger perceived time savings than tightly controlled observed tasks. DBT deliberately included observed sessions and applied conservative adjustments so that reported time savings would better reflect net effects rather than optimistic first impressions. The evaluation explicitly warns that short pilots and self‑reporting can overstate impact unless corroborated by observed or longitudinal data. (assets.publishing.service.gov.uk)

---

Key findings — satisfaction, time savings and the productivity gap​

Strong user satisfaction, concentrated benefits​

Satisfaction was high: 72% of DBT respondents reported being satisfied or very satisfied with Copilot and the department recorded a Net Promoter Score of 31, which the report described as a good outcome for a new digital service. Satisfaction clustered most strongly around text‑centric tasks — drafting, editing and summarising — and weaker for tasks such as scheduling and image generation. (assets.publishing.service.gov.uk)
Notable accessibility and inclusion benefits emerged: neurodiverse staff and non‑native English speakers were statistically more likely to report higher satisfaction, with qualitative interviews highlighting improvements in meeting accessibility, comprehension and confidence. These social outcomes are real and separate from pure productivity math. (assets.publishing.service.gov.uk)

Task‑level time savings — significant but uneven​

DBT’s diary analysis — after adjusting for unused outputs and novel tasks — produced task‑level mean time savings (hours per task) that were substantial for some activities:

• Drafting written documents: ~1.3 hours saved per task. (assets.publishing.service.gov.uk)
• Summarising research: ~0.8 hours saved per task. (assets.publishing.service.gov.uk)
• Transcribing/summarising meetings and searching for information: ~0.7 hours saved per task. (assets.publishing.service.gov.uk)

At the same time, some tasks took longer with Copilot: scheduling averaged ‑0.6 hours (i.e., added time), and image generation also tended to add time once unused and novel outputs were accounted for. (assets.publishing.service.gov.uk)
These per‑task gains are real for many users: in observed sessions Copilot users produced faster and often higher‑quality summaries and email drafts. But gains were very context dependent — for example, Excel data analysis sometimes became slower and poorer quality in observed tasks, offsetting wins elsewhere. (assets.publishing.service.gov.uk)

Satisfaction rose — measured productivity did not​

Despite positive user sentiment and measurable per‑task time savings, DBT concluded it did not find robust evidence that those savings produced department‑level productivity improvements during the short pilot window. Control‑group colleagues outside the pilot reported no visible change in overall output from participants, and the evaluation cautioned that small per‑task savings do not automatically convert to organisation‑level gains without changes in workflows, governance and long‑term measurement. (assets.publishing.service.gov.uk)
This departmental conclusion sits beside the GDS cross‑government experiment, which reported an average of 26 minutes per day saved across a much larger, cross‑department sample — a headline that highlights how scope and measurement frame change the story. Both findings are valid in their own terms: DBT used a conservatively adjusted, evidence‑led approach; GDS aggregated broader self‑reported survey data. (gov.uk)

---

Why the headlines diverge: measurement, scale and task mix​

Three primary reasons explain divergent narratives in media and public discussion:

• Scale and sample composition. DBT’s pilot (1,000 licences) was departmental and role‑specific; GDS’s experiment (20,000 licences) averaged across many departments and role mixes. Aggregation smooths variance and lifts averages in ways that can look like general productivity uplift. (assets.publishing.service.gov.uk, gov.uk)
• Metric choice. GDS emphasised a daily minutes‑saved average derived from self‑report surveys; DBT stressed adjusted per‑task hours from diaries and saw the need for observed validation. Self‑report tends to overstate time saved relative to observed timings. (assets.publishing.service.gov.uk, gov.uk)
• Task mix matters. Copilot’s strongest wins are in templated writing, summarisation and meeting notes. Organisations heavy in data‑analysis or specialised workflows may see smaller or even negative net effects if verification and correction overheads grow. (assets.publishing.service.gov.uk)

Media summaries that extract a single number (for example “26 minutes a day” or “users saved two weeks a year”) simplify a complex evidence base; the underlying official documents must be read to understand the boundaries of each claim. (gov.uk, ft.com)

---

Strengths and practical wins​

• Clear wins in communication‑heavy tasks. Copilot reliably speeds drafting, summarisation and meeting transcriptions when outputs are used as drafts and human verification is applied. (assets.publishing.service.gov.uk)
• Accessibility and inclusion benefits. Automated transcriptions and concise summaries helped neurodiverse users and staff for whom English is a second language, improving clarity and confidence. (assets.publishing.service.gov.uk)
• High user satisfaction. A strong NPS and broad acceptance indicate cultural receptivity — a necessary precondition for realizing any longer‑term productivity gains. (assets.publishing.service.gov.uk)

---

Risks, hidden costs and governance gaps​

• Verification overhead and hallucinations. DBT documented instances of hallucinations (confidently wrong outputs) that required human checking; the effort of verification can erase time savings. Where outputs feed downstream decisions without proper review, operational risk rises. (assets.publishing.service.gov.uk)
• Task‑specific degradation. In observed sessions, some data‑heavy tasks (notably Excel analysis) were slower or less accurate with Copilot — a direct counterexample to blanket productivity claims. (assets.publishing.service.gov.uk)
• Novel‑task paradox. Copilot made some work easy enough that staff performed additional, previously uncompleted tasks (novel tasks), increasing workload rather than reducing it. The DBT evaluation adjusted for this and found it materially affected net time savings. (assets.publishing.service.gov.uk)
• Environmental and procurement blind spots. Pilot participants raised concerns about the energy footprint of large language models; DBT flagged the lack of quantified environmental impact assessment as an open procurement risk. (assets.publishing.service.gov.uk)
• Vendor transparency and contract risk. Both DBT and GDS recommended contractual clarity on data use, model training, retention and environmental disclosure before large‑scale procurement. (assets.publishing.service.gov.uk, gov.uk)

---

What independent media and industry claims add — corroboration and caution​

Industry leaders have made broader productivity claims. For example, Microsoft CEO Satya Nadella publicly stated that up to 20–30% of Microsoft’s code is now written by AI in some projects — a claim widely reported by major outlets and indicative of radical workplace change in engineering contexts. Those corporate claims track with some firms’ aggressive adoption and reported cost savings, but they are not direct evidence that every organisation or role will see comparable gains. (cnbc.com, geekwire.com)
Independent reporting and commentary emphasise nuance: several outlets and analysts note that while AI can generate volume, it can also introduce a form of technical debt or “vibe coding” that requires engineering oversight, testing and remediation. These industry realities mirror DBT’s finding that human verification remains essential and that cost‑savings on paper can hide follow‑on remediation costs. (ft.com, windowscentral.com)

---

Practical guidance for IT leaders, procurement and M365 administrators​

The DBT and GDS evaluations converge on actionable recommendations for public‑sector IT and any organisation considering Copilot‑style deployments:

• Pilot deliberately and restrictively: start with a narrow set of high‑volume, low‑risk tasks such as meeting notes, templated emails and document summarisation. (assets.publishing.service.gov.uk, gov.uk)
• Measure the right things:
• Combine telemetry with timed observed tasks, not surveys alone.
• Convert minutes saved into financial terms for the pay bands being targeted.
• Factor in training and verification time when modelling ROI. (assets.publishing.service.gov.uk)
• Invest in training and human‑in‑the‑loop workflows:
• Provide hands‑on prompt engineering and verification training.
• Make self‑directed learning available — DBT found self‑led training correlated with higher satisfaction. (assets.publishing.service.gov.uk)
• Insist on vendor transparency and contractual safeguards:
• Explicit clauses about whether tenant data may be used to improve vendor models.
• Data residency, retention and DLP guarantees.
• Environmental disclosures or compute‑footprint commitments. (assets.publishing.service.gov.uk, gov.uk)
• Segment rollouts by role and workflow: deploy where verification overhead is low and benefits are concentrated; defer or tightly restrict use in sensitive data‑analysis roles until behaviour is validated. (assets.publishing.service.gov.uk)

---

What remains unproven and what needs longer measurement​

DBT’s cautionary conclusion is not a rejection of Copilot’s potential; it is a call for stronger, longer, and more narrowly designed measurement. The following aspects require follow‑up before organisations treat Copilot as a proven productivity multiplier:

• Longitudinal behaviour change: habit formation, prompt literacy and deeper UX adaptation occur over many months, not a truncated three‑month pilot.
• Organisation‑level ROI modelling that converts per‑task minutes into net financial outcomes after verification, training and remediation costs are included.
• Quantified environmental impact and life‑cycle assessments for large‑scale LLM usage.
• Independent third‑party audits of hallucination rates and failure modes in mission‑critical workloads. (assets.publishing.service.gov.uk, gov.uk)

Where managers or vendors point to single headline numbers, procurement teams should ask: “Which metric, which sample and what adjustment method produced this number?” Different answers imply very different expected outcomes.

---

Conclusion — measured optimism, not hype​

The DBT pilot demonstrates a pragmatic reality that many organisations will recognise: Copilot is useful and, for many people, satisfying — and it produces measurable time savings on specific tasks — but those savings do not automatically add up to department‑level productivity gains without follow‑through on governance, measurement and workflow redesign. (assets.publishing.service.gov.uk)
For IT leaders and Windows/M365 administrators the immediate imperative is operational discipline: pilot with purpose, measure conservatively, invest in verification and training, and require vendor transparency. When those levers are pulled correctly, the per‑task wins DBT documented can compound into real capacity improvements. When they are not, outputs that feel faster can generate hidden work and risk.
The debate now is not whether Copilot can help people — it can — but exactly how organisations will convert those human‑level wins into durable, accountable operational value. The DBT evaluation is a useful, evidence‑based checkpoint on that path. (assets.publishing.service.gov.uk, gov.uk)

---

Source: TechRepublic Microsoft Copilot Study in UK: No Evidence of Productivity Gains