The practical change for enterprises building agentic AI is not a new Microsoft server product; it is a warning that sizing infrastructure around GPU utilization alone can badly misread the workload. A new paper from researchers at Microsoft Azure and the University of Texas at Austin finds that agent workflows routinely alternate between model inference, orchestration, state management, and external tool execution, putting CPUs, memory, networking, and runtime scheduling onto the critical path alongside GPUs.
Network World, which first reported the findings on August 7, points to production telemetry from Azure and controlled experiments with open-source agent frameworks. The underlying preprint, Architectural Implications of Agentic AI Workflows, supplies the more consequential detail: this is not merely an argument for adding more CPU cores to GPU servers. It argues that static CPU/GPU ratios and a single undifferentiated host-core pool are a poor fit for workloads whose demand can be quiet for long stretches and then spike sharply when agents fan out into builds, tests, retrieval, database calls, or other tools.
For IT teams accustomed to treating an LLM service as an accelerator-heavy request/response system, the conclusion is clear: agent serving is a distributed application problem with inference inside it. The GPU remains essential, but it no longer describes the whole performance envelope.
The researchers collected a 24-hour trace of production agentic requests in Microsoft Azure, measuring time, tool activity, and CPU behavior across three host-side roles: scheduler, orchestrator, and runner. The scheduler dispatches inference, the orchestrator advances the workflow and routes messages or state, and the runner executes agent logic and invokes tools.
Their production result is unusually direct. Tool execution time was comparable to or greater than LLM inference time in more than 27% of requests. That finding changes the meaning of a familiar utilization dashboard: a GPU can appear underused not because the service is overprovisioned, but because an agent is waiting on a tool, a handoff, a database, a compilation step, or host-side control logic before it can make its next model call.
One example Azure request ran for nearly a minute and contained multiple LLM calls, three tool-discovery operations, and three executions across two tools. The paper’s controlled CORAL test went further: one run generated 580 LLM calls interleaved with 552 tool invocations. The point is not that every enterprise agent will make more than a thousand operations. It is that the atomic unit for capacity planning is no longer a prompt and completion; it is an unpredictable, data-dependent execution graph.
Microsoft’s own Azure architecture guidance independently reflects this operational reality. Its reference design for large multi-agent deployments includes Azure OpenAI or Foundry hosting, Redis for conversational state, AI Search, Application Insights, Azure Monitor, Log Analytics, and external services. Its current orchestration guidance also cautions that multi-agent decision-making and flow-control overhead can outweigh the benefit of splitting a task among agents. In other words, the research measures a systems cost that Microsoft’s platform documentation already treats as a deployment concern.
The paper does leave a material gap: Microsoft and the authors do not identify the Azure agent services involved, the number of requests in the trace, the customer mix, the model families, or the geographic footprint. “Production” establishes that the behavior was observed outside a lab, but it does not make the reported distribution a universal Azure capacity ratio. Enterprises should regard the 27% figure as evidence that tool time can dominate at meaningful scale, not as a benchmark to transpose into a procurement worksheet.
This is the failure mode for a conventional “right-size to average utilization” strategy. A server provisioned for the typical period leaves agent work waiting during short tool bursts. A server provisioned permanently for the burst spends much of its life with spare cores. The same mismatch appears on the accelerator side when agent roles use different models or are active at different times.
In the paper’s Owl workload, four of eight GPUs handling the busiest text and vision roles repeatedly approached peak power while the other four spent much of the run near 20% of peak power. Adding another identical eight-GPU node would not solve that imbalance; it would reproduce it. The relevant question is whether model copies, agent placement, and state can be consolidated safely so that busy and idle roles complement rather than isolate one another.
That has immediate implications for on-premises AI projects. A cluster built around one model per GPU, dedicated GPU reservations per agent role, and broadly shared CPU worker pools may be easier to deploy, but it may strand the exact capacity for which the organization paid. Conversely, aggressive consolidation without tail-latency protection can turn a burst of tool calls into a stalled workflow.
The researchers’ conclusion is more nuanced than “buy heterogeneous hardware.” They say a machine needs to respond to the type of workflow and its current load. A sequential agent whose roles activate unevenly can yield idle GPU capacity to other work. A heavily parallel workflow may keep all roles busy, making GPU harvesting counterproductive even when consolidation still improves caching or model-weight sharing.
The reported numbers are large. In the test setup, CPU harvesting increased host CPU utilization by 30% and recovered 95% of a colocated workload’s standalone throughput while holding agent slowdown below 3%. In an Owl configuration, GPU consolidation freed one-third of GPUs, increased generation throughput by 82%, completed 22% more tasks per hour, and reduced tail latency by 2.5 times. At a fixed GPU budget, the same technique reportedly increased throughput by 106% and cut tail latency by 3.8 times.
Those figures deserve careful reading. They are prototype evaluation results, not service-level commitments from Azure, Microsoft Foundry, NVIDIA, AMD, or any server vendor. The controlled experiments used a 96-core AMD EPYC 7V12 system with eight NVIDIA A100 GPUs, vLLM serving instances, and four named frameworks: SWE-Agent, Trae, CORAL, and Owl. The results also vary materially by workflow: the paper says GPU harvesting helps the unevenly active Owl case, while attempting to harvest GPUs from parallel CORAL can cost more throughput than the capacity it saves.
No announcement accompanies the paper for an “Agora” product, a new Azure VM family, a server reference design, pricing, availability date, or supported migration path for existing AI clusters. The research should therefore inform architecture decisions today, but it should not be mistaken for a purchasing announcement.
At higher agent concurrency, the problem became visible even when total CPU utilization remained low. For SWE-Agent, involuntary context switches rose from 71 per second at one concurrent task to 660 per second at 32. More simultaneous agents did not simply improve GPU batching; they created more scheduler, dispatch, state-management, and context-switching work.
For Windows-centric enterprise development teams, this maps cleanly to familiar server engineering practice. The answer is not a simplistic “make every process high priority” policy. It is to identify the different service roles, isolate latency-sensitive control paths from bursty runners where appropriate, preserve affinity for workloads that benefit from it, and instrument the whole request path rather than only the model endpoint.
Microsoft Foundry’s tracing documentation supports that approach. It records inputs, outputs, tool use, retries, latencies, token consumption, and costs, with OpenTelemetry-based spans for agent-to-agent interactions, planning, orchestration, state management, and tool execution. That telemetry carries a governance consequence as well: prompts, tool arguments, and tool results can contain sensitive data. Foundry explicitly advises treating traces as production telemetry, applying access controls and retention policies, and redacting secrets and personal data before storage.
A useful first review should establish:
For IT teams accustomed to treating an LLM service as an accelerator-heavy request/response system, the conclusion is clear: agent serving is a distributed application problem with inference inside it. The GPU remains essential, but it no longer describes the whole performance envelope.
Azure’s production data shows tool work is not incidental
The researchers collected a 24-hour trace of production agentic requests in Microsoft Azure, measuring time, tool activity, and CPU behavior across three host-side roles: scheduler, orchestrator, and runner. The scheduler dispatches inference, the orchestrator advances the workflow and routes messages or state, and the runner executes agent logic and invokes tools.Their production result is unusually direct. Tool execution time was comparable to or greater than LLM inference time in more than 27% of requests. That finding changes the meaning of a familiar utilization dashboard: a GPU can appear underused not because the service is overprovisioned, but because an agent is waiting on a tool, a handoff, a database, a compilation step, or host-side control logic before it can make its next model call.
One example Azure request ran for nearly a minute and contained multiple LLM calls, three tool-discovery operations, and three executions across two tools. The paper’s controlled CORAL test went further: one run generated 580 LLM calls interleaved with 552 tool invocations. The point is not that every enterprise agent will make more than a thousand operations. It is that the atomic unit for capacity planning is no longer a prompt and completion; it is an unpredictable, data-dependent execution graph.
Microsoft’s own Azure architecture guidance independently reflects this operational reality. Its reference design for large multi-agent deployments includes Azure OpenAI or Foundry hosting, Redis for conversational state, AI Search, Application Insights, Azure Monitor, Log Analytics, and external services. Its current orchestration guidance also cautions that multi-agent decision-making and flow-control overhead can outweigh the benefit of splitting a task among agents. In other words, the research measures a systems cost that Microsoft’s platform documentation already treats as a deployment concern.
The paper does leave a material gap: Microsoft and the authors do not identify the Azure agent services involved, the number of requests in the trace, the customer mix, the model families, or the geographic footprint. “Production” establishes that the behavior was observed outside a lab, but it does not make the reported distribution a universal Azure capacity ratio. Enterprises should regard the 27% figure as evidence that tool time can dominate at meaningful scale, not as a benchmark to transpose into a procurement worksheet.
Static provisioning wastes capacity — and can still miss latency targets
The research identifies a counterintuitive pattern. Agent workloads can leave CPU and GPU capacity idle on average while still hitting near-saturation at particular workflow stages. In a Trae coding-agent experiment, host CPU use remained around an 11% median during sequential work, then climbed near 100% when stages released multiple builds and test runs.This is the failure mode for a conventional “right-size to average utilization” strategy. A server provisioned for the typical period leaves agent work waiting during short tool bursts. A server provisioned permanently for the burst spends much of its life with spare cores. The same mismatch appears on the accelerator side when agent roles use different models or are active at different times.
In the paper’s Owl workload, four of eight GPUs handling the busiest text and vision roles repeatedly approached peak power while the other four spent much of the run near 20% of peak power. Adding another identical eight-GPU node would not solve that imbalance; it would reproduce it. The relevant question is whether model copies, agent placement, and state can be consolidated safely so that busy and idle roles complement rather than isolate one another.
That has immediate implications for on-premises AI projects. A cluster built around one model per GPU, dedicated GPU reservations per agent role, and broadly shared CPU worker pools may be easier to deploy, but it may strand the exact capacity for which the organization paid. Conversely, aggressive consolidation without tail-latency protection can turn a burst of tool calls into a stalled workflow.
The researchers’ conclusion is more nuanced than “buy heterogeneous hardware.” They say a machine needs to respond to the type of workflow and its current load. A sequential agent whose roles activate unevenly can yield idle GPU capacity to other work. A heavily parallel workflow may keep all roles busy, making GPU harvesting counterproductive even when consolidation still improves caching or model-weight sharing.
Agora is a prototype result, not an Azure product roadmap
To test their design principles, the researchers built a commodity-server prototype called Agora. It dynamically lends idle runner CPU capacity to colocated throughput jobs, while attempting to protect agent tail latency when a tool burst arrives. It also separates scheduler, orchestrator, and runner work into distinct CPU pools, applies task affinity to reduce cache and branch-predictor disruption, and consolidates agents on fewer GPUs when their activity patterns and state permit it.The reported numbers are large. In the test setup, CPU harvesting increased host CPU utilization by 30% and recovered 95% of a colocated workload’s standalone throughput while holding agent slowdown below 3%. In an Owl configuration, GPU consolidation freed one-third of GPUs, increased generation throughput by 82%, completed 22% more tasks per hour, and reduced tail latency by 2.5 times. At a fixed GPU budget, the same technique reportedly increased throughput by 106% and cut tail latency by 3.8 times.
Those figures deserve careful reading. They are prototype evaluation results, not service-level commitments from Azure, Microsoft Foundry, NVIDIA, AMD, or any server vendor. The controlled experiments used a 96-core AMD EPYC 7V12 system with eight NVIDIA A100 GPUs, vLLM serving instances, and four named frameworks: SWE-Agent, Trae, CORAL, and Owl. The results also vary materially by workflow: the paper says GPU harvesting helps the unevenly active Owl case, while attempting to harvest GPUs from parallel CORAL can cost more throughput than the capacity it saves.
No announcement accompanies the paper for an “Agora” product, a new Azure VM family, a server reference design, pricing, availability date, or supported migration path for existing AI clusters. The research should therefore inform architecture decisions today, but it should not be mistaken for a purchasing announcement.
CPU contention is a scheduling problem as much as a core-count problem
The most overlooked finding is beneath the utilization charts. The researchers observed that mixing many agent tasks freely across shared cores damages microarchitectural locality—the ability of a process to benefit from warm caches and learned branch-prediction state. In the controlled workloads, three of the four frameworks lost 43% to 47% of CPU pipeline slots to backend stalls, while low instructions-per-cycle and elevated cache misses indicated host cores were spending considerable time waiting rather than executing useful work.At higher agent concurrency, the problem became visible even when total CPU utilization remained low. For SWE-Agent, involuntary context switches rose from 71 per second at one concurrent task to 660 per second at 32. More simultaneous agents did not simply improve GPU batching; they created more scheduler, dispatch, state-management, and context-switching work.
For Windows-centric enterprise development teams, this maps cleanly to familiar server engineering practice. The answer is not a simplistic “make every process high priority” policy. It is to identify the different service roles, isolate latency-sensitive control paths from bursty runners where appropriate, preserve affinity for workloads that benefit from it, and instrument the whole request path rather than only the model endpoint.
Microsoft Foundry’s tracing documentation supports that approach. It records inputs, outputs, tool use, retries, latencies, token consumption, and costs, with OpenTelemetry-based spans for agent-to-agent interactions, planning, orchestration, state management, and tool execution. That telemetry carries a governance consequence as well: prompts, tool arguments, and tool results can contain sensitive data. Foundry explicitly advises treating traces as production telemetry, applying access controls and retention policies, and redacting secrets and personal data before storage.
Measure the workflow before replacing the hardware
The immediate action is a capacity-planning reset, not a forklift refresh. Teams deploying Microsoft Agent Framework, Semantic Kernel, Foundry Agent Service, LangGraph, CrewAI, or custom .NET/Python orchestrators should trace end-to-end runs and break latency down into inference, orchestration, tool execution, queueing, retries, and state transfers. Measure percentile behavior, especially simultaneous tool bursts and long-running outliers, rather than relying on average CPU or GPU utilization.A useful first review should establish:
- Whether the agent’s workflow is sequential, parallel, or a mixture, because the structure determines whether demand arrives smoothly or in bursts.
- Whether separate agent roles duplicate model weights and KV caches on GPUs that are rarely active at the same time.
- Whether runners, orchestrators, and model schedulers contend for the same CPU pools, storage, network paths, or service quotas.
- Whether tracing data exposes credentials, regulated content, or customer records when tool calls and agent state are collected for debugging.
- Whether a simpler single-agent or deterministic workflow can meet the business requirement before a multi-agent design multiplies coordination cost.
References
- Primary source: Network World
Published: August 7, 2026 at 3:26 AM UTC
Agentic AI could force a rethink of enterprise AI server design, researchers say | Network World
Microsoft Azure and University of Texas researchers found that multi-step AI workflows create CPU-GPU bottlenecks that conventional inference infrastructure struggles to handle efficiently.www.networkworld.com
- Related coverage: learn.microsoft.com
Monitor, Evaluate, and Operate Multi-Agent AI Solutions in Azure - Training | Microsoft Learn
Learn how to operate production multi-agent solutions with comprehensive visibility, systematic quality assurance, cost control, and robust incident response. Learners advance from single-application monitoring and evaluation experiments to distributed multi-agent observability, LLM-as-judge...learn.microsoft.com - Related coverage: learn.microsoft.com
Dynamic AI Agents at Scale Pattern - Azure Architecture Center | Microsoft Learn
Learn how to dynamically select and orchestrate AI agents from a pool by using Microsoft Foundry, Azure AI Search, and Azure OpenAI to build multiagent systems.learn.microsoft.com - Related coverage: devblogs.microsoft.com
Build agents you can trust across any framework with open evals and a control standard | Microsoft Foundry Blog
Learn how Microsoft helps developers build trustworthy AI agents with open evaluations, portable runtime controls, production observability, and security workflows that work across frameworks.devblogs.microsoft.com - Related coverage: techcommunity.microsoft.com
- Related coverage: conf.researchr.org
An Agentic Framework for Triaging Incidents in Production Cloud Infrastructure (FSE 2026 - Industry Papers) - FSE 2026
The ACM International Conference on the Foundations of Software Engineering (FSE) is an internationally renowned forum for researchers, practitioners, and educators to present and discuss the most recent innovations, trends, experiences, and challenges in the field of software engineering. FSE...conf.researchr.org
- Related coverage: techcommunity.microsoft.com
- Related coverage: ust.com