That is the useful point in a new Conversation analysis linking two 2025 research projects with the ongoing Mobley v. Workday employment-discrimination case in California. The Workday litigation is a live example of the distinction. It is not a court finding that Workday’s systems discriminated; the plaintiffs allege that the vendor’s automated recruitment and screening tools produced unlawful disparate impacts based on race, age, disability, and later gender. Workday has denied the allegations, saying its recruiting technology considers job qualifications rather than protected characteristics.
But the case also demonstrates why “our model does not use race, age, or disability as an input” is not, by itself, an adequate operational answer. A screening system can be built without an explicit protected-trait field and still inherit patterns from prior hiring decisions, job-history proxies, geography, education, gaps in employment, wording conventions, or labels chosen by people assembling the training and evaluation data.
The Workday case is about accountability, not a rogue algorithm
The claims against Workday date to February 2023, so this is not a newly filed lawsuit. What has kept it in focus is the court’s willingness to let key age-discrimination claims proceed and the continued fight over evidence concerning the company’s AI screening and bias-testing practices.
Court filings in Mobley v. Workday portray Workday’s technology as a system that can evaluate, rank, screen, and reject applicants across a large employer customer base. The plaintiffs argue that this role gives Workday meaningful control over access to jobs, rather than making it a detached software supplier. Workday disputes that characterization along with the discrimination claims.
The allegations remain allegations. That is an important boundary, particularly for administrators who need evidence rather than headlines. Yet the dispute has practical relevance even before a final judgment: when a vendor provides a recommendation, score, shortlist, or automated disposition, an employer cannot treat the tool as a neutral black box simply because the vendor trained it elsewhere.
The EEOC and Department of Justice have already warned employers that employment tools using algorithms or AI must comply with federal civil-rights law, including disability-related obligations under the Americans with Disabilities Act. In other words, procurement does not transfer accountability. A company can contract for the model, the workflow, and the hosting, but not for the legal and operational consequences of using the result to limit a person’s opportunity.
For enterprise IT, the immediate takeaway is that automated recommendations must be treated as decision infrastructure, not productivity software. A Copilot-generated interview summary, a candidate-ranking agent, or a workflow that suggests who should advance may look advisory on a process diagram. If reviewers routinely accept the suggestion without meaningful scrutiny, the system is functioning as a decision-maker in practice.
The Copilot study is a warning, not a benchmark of today’s product
The Conversation article cites “What Does a Software Engineer Look Like? Exploring Societal Stereotypes in LLMs,” a 2025 study by Muneera Bano, Hashini Gunatilake, and Rashina Hoda. The researchers used GPT-4 and Microsoft Copilot in a simulated software-engineering recruitment exercise involving 300 candidate profiles and four roles ranging from junior engineer to lead engineer.
They report that both systems preferred male and white profiles, especially for senior positions, and that generated images of preferred candidates tended toward lighter skin, younger appearance, and slimmer body types. The study deliberately created paired profiles that changed gender while holding other attributes constant, alongside gender-neutral versions, to expose whether identity markers changed recommendations.
Its results should not be overstated. It was an exploratory, prompt-based study of the versions and image-generation capabilities available in 2025; it is not an audit of every present-day Microsoft Copilot product, tenant configuration, grounding source, or hiring product. GPT-4 itself has since been retired. A controlled research result also cannot establish that a particular enterprise deployment will produce the same outcomes.
Still, dismissing the work because the models have changed would miss its strongest finding. The study did not identify a single bad line of code that can simply be removed. It showed that a realistic recruitment prompt can pull familiar social stereotypes through both text selection and image generation. That is a warning about system design: input framing, job descriptions, prompt wording, evaluation criteria, and the chosen outputs can all carry bias into a workflow before the model responds.
The image component deserves particular attention. If an organization uses generative AI to create recruiting materials, persona illustrations, training examples, or executive-facing candidate summaries, visual output can quietly reinforce a narrow picture of who belongs in technical leadership. Such images may never make a formal hiring decision, yet they can influence the assumptions of the humans who do.
Incident data points to a lifecycle failure
A separate 2025 paper, “AI for All: Identifying AI Incidents Related to Diversity and Inclusion,” manually examined incidents cataloged in the AI Incident Database and the AI, Algorithmic, and Automation Incidents and Controversies database. Published in the Journal of Artificial Intelligence Research, it found that nearly half of the incidents reviewed involved diversity and inclusion issues, with racial, gender, and age discrimination appearing most often.
The figure is not a measure of how often all AI systems discriminate. Incident databases contain reported and documented failures, not a random sample of every model in use. Their value lies elsewhere: they show recurring failure modes after systems meet real people, organizations, and uneven social conditions.
The researchers trace problems across the lifecycle: insufficiently diverse data, neglected inclusion requirements during design, weak evaluation, and deployment practices that do not account for how a system will be used. That pattern supports the Conversation article’s core claim that technical mitigation alone is incomplete.
Rebalancing a dataset, reducing a statistical disparity, or adding a content filter can be worthwhile. None answers the preceding questions: Was the product necessary for this high-impact task? Did the team identify the populations who could be harmed? Were affected people able to challenge the result? Did the buyer test the vendor’s claims in its own environment? Who can stop the system if an incident emerges?
Those are governance questions, and they cannot be solved by the model team alone.
Microsoft’s own guidance sets a higher operational bar
Microsoft’s current Responsible AI guidance for enterprise agents is more demanding than a one-time pre-launch review. It calls for human approval where an agent takes consequential actions affecting people, money, or compliance; clear escalation paths for sensitive or ambiguous cases; and continuous monitoring because models, data, use patterns, and regulations can change after deployment.
That guidance aligns with the National Institute of Standards and Technology’s AI Risk Management Framework. NIST separates AI risk work into four continuing functions: Govern, Map, Measure, and Manage. The structure matters because it prevents an organization from treating fairness as a single score on a model-validation report.
“Govern” assigns responsibility and policies. “Map” identifies the intended context, potential impacts, affected groups, and third-party dependencies. “Measure” evaluates identified risks and documents the results. “Manage” prioritizes and responds to the risks found. NIST specifically calls for defined human-oversight processes, evaluation of fairness and harmful bias, ongoing monitoring, and diverse, multidisciplinary input into decision-making.
For Windows and Microsoft 365 administrators, this should translate into controls that can be operated rather than values stated in a policy. Any team deploying Copilot Studio agents or Azure-based AI into HR, employee management, finance, education, benefits, security triage, or customer eligibility should be able to produce an auditable record of:
- The precise action the AI can recommend, automate, or block, including the circumstances in which a person must approve it.
- The data sources, access permissions, retention rules, and proxy variables that can influence the output.
- Tests using realistic edge cases, including people who sit across multiple protected characteristics rather than one demographic category at a time.
- A mechanism for users and affected people to report suspected harm, receive a human review, and obtain correction where appropriate.
- Named owners for model changes, grounding-data changes, prompt changes, incident response, and the authority to suspend the workflow.
The hard part is not writing this checklist. It is making the controls survive commercial pressure to deploy quickly and operational pressure to let staff accept AI recommendations automatically.
“Human in the loop” must mean authority to disagree
A reviewer who sees only a score, a terse explanation, and a button labeled “approve” is not meaningful oversight. They are a rubber stamp with legal exposure. Human review needs enough information, time, training, and institutional permission to challenge an AI output.
That includes measuring overrides. If reviewers almost never overrule a system, administrators should determine whether the tool is genuinely excellent, whether staff are over-trusting it, or whether the interface makes dissent costly. If overrides cluster around particular roles, regions, language groups, or disability-accommodation situations, that is evidence worth investigating—not noise to smooth away.
The Workday case has not yet resolved the plaintiffs’ claims, and the Copilot research cannot be read as a verdict on every current Microsoft model. Together, however, they support a narrower and more useful conclusion: when AI affects a person’s access to work, services, money, or opportunity, an organization must be able to explain the human choices surrounding the system—not merely the technical properties of the model inside it.