Elastic’s new integration with Azure AI Foundry brings real‑time observability to agentic AI and large language model (LLM) workloads, delivering pre‑built dashboards for token usage, latency, cost tracking, and content filtering inside the Elastic Observability environment — currently available as a technology preview.
Background
The rapid adoption of agentic AI — systems that orchestrate multiple steps, tools, and model calls to achieve a goal — has turned basic model telemetry into a first‑class operational requirement. Site reliability engineers (SREs), developers, and compliance teams now need continuous insight into how models consume tokens, how long responses take, which downstream systems are invoked, and whether outputs meet policy and regulatory constraints.
Elastic’s Azure AI Foundry integration stitches those needs into the Elastic Observability stack. The integration exposes metrics and logs produced by Azure AI Foundry and presents them through ready‑made visualizations so teams can spot cost drivers, performance bottlenecks, and compliance exceptions without stitching together dashboards from multiple systems.
This article explains what the integration delivers, how it works in practice, the operational benefits, and the risks and caveats organisations should consider before deploying agentic AI at scale.
What the integration delivers
Elastic’s integration focuses on four core observability concerns for LLM and agentic workloads:
- Model usage: Track token consumption (input, output, total) and request counts to understand direct cost drivers.
- Performance telemetry: Surface latency metrics such as time‑to‑first‑token, time‑to‑last‑byte, and normalized per‑token latency so teams can detect responsiveness regressions.
- Cost and capacity indicators: Show provisioning utilization and cost signals that indicate throttle risk or overspend.
- Content and policy monitoring: Provide filters and alerts for risky content or outputs that may violate policy or compliance requirements.
These capabilities are exposed via pre‑configured dashboards designed for rapid troubleshooting and ongoing optimisation. Real‑time streams of metrics and gateway logs let engineers correlate spikes in token usage with particular calls, agent actions, or external tool integrations, giving context that basic logs or billing reports cannot provide.
Santosh Krishnan, general manager of Observability & Security at Elastic, frames the proposition succinctly:
visibility into model performance and cost is foundational to scaling agentic applications without compromising reliability or compliance. Similarly, Microsoft Azure’s product leadership positions the integration as delivering the operational clarity required to run models and agents in production.
How it works: telemetry, pipelines, and dashboards
Data collection and ingestion
The integration leverages Azure’s built‑in telemetry and Elastic’s ingestion mechanisms to capture relevant signals:
- Azure AI Foundry emits logs and metrics (audit events, request/response records, gateway logs, and model metrics).
- Diagnostic settings and API Management may be used to route gateway logs and request/response data to telemetry destinations such as Event Hubs.
- Elastic’s integration consumes these telemetry streams (for example, via Azure Event Hub) and normalises them into Elastic’s data model so dashboards and alerts can be built quickly.
A few practical technical notes are important here. Azure AI Foundry exposes a collection of metrics that are explicitly useful for observability: token counts (input, output, total), request counts, latency measurements, and provisioned utilization figures for provisioning‑based deployments. Metrics often arrive at a five‑minute granularity by default, which is adequate for trend detection and many operational use cases but may not be truly sub‑second real‑time for high‑frequency workloads.
Dashboard capabilities
Out of the box, Elastic provides pre‑configured dashboards and visualisations that typically include:
- Time series charts for token consumption and requests.
- Latency percentiles and per‑token latency metrics.
- Heatmaps linking agents/agents’ actions to latency and errors.
- Cost‑oriented views that map token consumption and provisioning to estimated spend.
- Filters and content‑safety dashboards to flag potentially non‑compliant outputs.
These dashboards are intentionally designed for both SREs doing rapid incident response and product teams tracking ongoing model performance and cost.
Alerts and correlation
Elastic’s platform enables alerting rules and correlation across data sources. That means an alert can be created to notify on sudden jumps in token usage, correlated with a spike in 429 throttling responses or increases in time‑to‑first‑token. Alerts can be forwarded to incident management tools or integrated with on‑call rotations.
Why this matters for SREs and developers
Agentic AI is effectively a distributed, orchestrated application that calls models, APIs, databases, and external systems. The traditional separation between application monitoring and cloud billing has left gaps that cause surprises in production:
- Uncontrolled token consumption can blow budgets overnight.
- Latency at the model layer can cascade, causing timeouts further down the stack.
- Content‑policy violations may expose organisations to compliance or legal risk.
- Provisioned deployments may be underutilised or exceed capacity, leading to throttles or unnecessary spend.
By combining metrics, logs, and pre‑built visual analytics, Elastic’s integration aims to give teams a unified operational picture. The result is a practical advantage for organisations that must keep agentic AI systems reliable, performant, and cost‑efficient in production.
Use cases and practical scenarios
1. Rapid incident triage
An agent repeatedly fails to complete a task. With the integration, an SRE can:
- Check recent token usage to see if prompt size or output size spiked.
- Inspect latency metrics (time‑to‑first‑token and time‑to‑last‑byte) to determine if a model or endpoint is slow.
- Correlate gateway logs to see whether a 429 or 500 error was returned upstream or whether downstream services failed.
This reduces mean time to resolution by giving a single pane of glass for both observability and root‑cause evidence.
2. Cost control and optimisation
Teams can monitor token usage broken down by model, agent, or workspace and map that consumption to budget owners. Alerts can be created for anomalous increases in consumption, enabling rapid response before spend escalates.
3. Compliance and safety monitoring
Content filters and logging of policy‑relevant signals let compliance teams track outputs that may trigger human review. Where regulations require retention and auditability, telemetry can be captured to provide traceable evidence of what an agent produced and why.
4. Capacity planning
Provisioned utilization metrics show percentage utilisation of provisioned throughput units. When utilisation approaches thresholds that trigger throttling, teams can either scale up or rearchitect to smooth load and avoid 429s.
Strengths: what this integration gets right
- Unified telemetry for model workloads. Bringing LLM metrics into an established observability platform reduces the integration burden and shortens time to insights.
- Pre‑built dashboards accelerate time to value. SREs and developers can begin troubleshooting without lengthy custom dashboard work.
- Correlation across signals. The integration lets teams correlate model behaviour with other traces and logs in Elastic, enabling faster root cause analysis.
- Operational clarity on cost drivers. Token usage and provisioning visibility help teams manage budgets and identify runaway consumption patterns.
- Support for agentic AI workflows. The dashboards and metrics are designed around the needs of multi‑step agents, not only single LLM calls.
Risks, caveats, and technical limitations
No observability solution is a silver bullet. Organisations should be aware of several important limitations and risks.
Data privacy and logging inputs/outputs
By default, Azure AI Foundry’s native logging does not necessarily record full request inputs and outputs. Capturing prompt inputs and model outputs often requires additional diagnostic setup, such as routing API Management gateway logs or enabling richer request/response logging. That creates two related concerns:
- Privacy and compliance risk: Logging raw prompts can capture PII or sensitive business data. Organisations must implement strict redaction, retention, and access controls before ingesting raw prompt content into Elastic.
- Storage and cost: Storing full interactions at scale can create large volumes of data with associated storage costs and index management challenges.
Observability granularity and latency
Metrics are useful for trends and many incident types, but some metrics are provided at multi‑minute granularity (for example, five‑minute timegrains). For extremely latency‑sensitive use cases or for microsecond‑level troubleshooting, this may be insufficient. Teams must design for the level of granularity they need, potentially complementing these metrics with application‑level tracing where required.
Vendor and model diversity
Azure AI Foundry provides access to many models via a catalog, but model behaviour varies widely. Observability that shows token counts and latency cannot, by itself, detect model hallucination or semantic drift; higher‑order checks (validation pipelines, grounding controls, human‑in‑the‑loop checks) remain necessary.
Reliance on correct telemetry configuration
The usefulness of the integration depends heavily on correct diagnostic configuration in Azure (for example, enabling the right diagnostic categories and streaming them to Event Hubs). Misconfiguration results in blind spots and incomplete dashboards — a classic “observability as configured” problem.
Security of telemetry pipelines
Telemetry pipelines that carry prompts, outputs, and agent activity are attractive attack surfaces. They must be protected with strong encryption, least‑privilege IAM, and monitoring for suspicious access to the telemetry store.
Operational checklist: deploying responsibly
To deploy this integration effectively and safely, organisations should treat observability as a program, not a checkbox. Recommended steps:
- Inventory data and sensitivity: classify prompt patterns and outputs that may contain PII, IP, or regulated data.
- Configure telemetry minimums: enable metrics (token usage, latency, requests) first; avoid logging raw prompts until redaction & access controls are in place.
- Set retention and access policies: implement short default retention for raw content with strict RBAC and audit logging.
- Build alerts for cost and throttling: create alerts that combine token usage spikes with provisioned utilization to detect runaway consumption.
- Implement content safety pipelines: couple content filters with human review workflows for high‑risk outputs.
- Test incident workflows: run tabletop exercises to validate that dashboards, alerts, and runbooks enable fast remediation.
- Monitor the observability stack itself: instrument Elastic and the ingestion pipeline for capacity, latency, and error metrics to avoid blind spots.
This sequence balances visibility with privacy and cost management.
Integration architecture — a closer look
A practical implementation pattern looks like this:
- Azure AI Foundry → Azure Monitor / Diagnostic settings → Event Hub (or storage) → Elastic ingestion pipeline (metricbeat/logstash/ingest) → Elastic Observability dashboards & alerting.
Key configuration points include diagnostic categories to collect, Event Hub throughput provisioning, and the mapping of Azure metric namespaces to Elastic index patterns. For provisioning‑based deployments, watch the utilization metric (percent provisioned consumption) to avoid throttling.
How this compares to other observability approaches
Traditional APM and logging tools were designed for web apps and microservices. LLM observability demands a few additional capabilities:
- Token accounting and model‑specific cost metrics.
- Per‑token latency normalization.
- Content safety filters that understand natural language outputs.
Elastic’s approach brings these LLM‑specific metrics into a general purpose observability platform. The advantage is correlation across systems and a mature alerting/visualisation layer. The tradeoff is that organisations must ensure Elastic’s indices and pipelines are configured for the volume and sensitivity of AI telemetry.
Other specialised vendors offer deep model‑centric observability (for example, drift detection, hallucination scoring, and semantic validation) — often complemented by Elastic’s broader observability if an organisation values a centralized monitoring architecture.
Governance, compliance, and auditability
Agentic AI running in regulated environments (finance, healthcare, government) raises clear governance requirements. Observability must support audit trails, explainability, and retention policies that align with regulation.
- Maintain immutable audit logs of agent actions where required.
- Implement robust policy enforcement (content filters, escape hatches, and human escalation).
- Ensure telemetry access is logged and auditable to support compliance reviews.
Because Azure’s default logging choices may not capture all event details, governance teams should coordinate with engineering early to define what telemetry is necessary for audit and compliance, then implement secure pipelines to capture it.
Practical limitations to be aware of today
- Tech preview status: The integration is currently distributed as a technology preview. Early access is valuable for testing, but organisations should not assume feature parity with fully supported GA releases. Plans, SLAs, and supported upgrade paths may differ until general availability.
- Granularity versus cost: Fine‑grained telemetry provides better troubleshooting but increases storage and ingest costs. Balancing telemetry fidelity against cost is essential.
- Model‑agnostic vs model‑aware insights: Token counts and latency are model‑agnostic and broadly useful. Semantic errors, hallucinations, or task correctness require specialised validation layers outside pure telemetry.
These points should temper expectations about what observability alone can solve.
Recommendations for SREs, developers, and security teams
- Begin with metrics, not raw prompts. Enable token and latency collection early; add request/response logging only after privacy and redaction policies are validated.
- Treat cost telemetry like a first‑class SLO. Define budget SLOs for token consumption, alert on deviations, and map spend to product owners.
- Use correlation to prioritize incidents. An uptick in token usage matched with a rise in per‑token latency and external API error rates is a clear signal for triage focus.
- Automate mitigation where safe. For example, implement rate limits or fallback routing to cheaper models when provisioning utilization spikes.
- Invest in content validation and scoring. Observability helps detect symptoms; correctness checks and semantic validators are needed to prevent harmful outputs.
Future outlook: observability for an increasingly agentic world
As agentic systems become more prevalent, observability will shift from passive dashboards to active control planes: systems that can automatically reroute calls, throttle agents that behave anomalously, or push emergency prompts to human overseers. The Elastic and Azure AI Foundry integration is a logical step toward that future, combining infrastructure telemetry with model telemetry in a way that supports both operational and governance needs.
Expect three major developments over the next 12–24 months:
- Tighter runtime controls that let platforms enforce token budgets and model selection dynamically.
- Richer model diagnostics (semantic correctness scores, hallucination detectors) integrated into observability flows.
- Standardised telemetry schemas for agentic activity that make cross‑platform monitoring and tooling easier.
Conclusion
Elastic’s Azure AI Foundry integration fills a practical gap: it provides SREs, developers, and compliance teams with structured, near‑real‑time telemetry tailored for LLMs and agentic AI. The integration’s strengths lie in unified telemetry, pre‑built dashboards, and correlation capabilities that accelerate incident response and cost governance.
However, observability is just one pillar of production readiness. Privacy controls for prompt logging, telemetry configuration choices, and semantic validation pipelines are all essential complements. Organisations should adopt a deliberate rollout: start with metric collection and cost alerts, harden data governance before ingesting raw prompts, and build automated mitigations for clear operational risks.
When used thoughtfully, the integration can substantially reduce the friction of scaling agentic AI into production. It gives teams the tools to watch what models do, quantify what they cost, and act when performance or compliance drifts — capabilities that are quickly becoming mandatory as AI moves from experiments to mission‑critical services.
Source: SecurityBrief Australia
Elastic integrates with Azure AI Foundry for real-time monitoring