The warning comes from Search Engine Journal’s commentary on DataDome’s July report, but the underlying figures are DataDome’s own telemetry: 17.7 billion AI-agent requests from April through June, up 45 percent from 12.2 billion in the prior quarter. DataDome says Meta-ExternalAgent made 5.3 billion requests on its network and Meta-WebIndexer made 3.75 billion, together exceeding half of the AI traffic it classified during the quarter.
Those figures are not a census of the web. DataDome sells bot-defense services, and its data represents more than 400 enterprises using its network rather than every site on the internet. Still, its direction of travel has been independently reported by Digiday and TechRadar, and the operational point does not depend on treating its customer base as universal: organizations that make crawler policy decisions from headlines about OpenAI or Google may be measuring the wrong traffic.
Meta’s two agents should not share one access rule
DataDome separates Meta’s automated traffic into two identities with materially different stated purposes. Meta-ExternalAgent is described as a collector used to retrieve public web content for Meta AI model training and improvement. Meta-WebIndexer is described as an indexer intended to keep Meta AI responses current.
That distinction should matter to anyone who runs an externally reachable application. A crawler that indexes a public documentation portal so an AI service can answer a current question is operationally different from a bulk collector that reads the same material for model development. Treating both as “Meta AI” and allowing or blocking them together may be convenient, but it is not a policy.
Search Engine Journal frames this as a publishing-economics problem: Google is receiving the pressure because publishers historically expected Googlebot crawling to translate into visitors, while Meta’s products have not been built around sending readers off-platform. That comparison explains the public debate, but it can obscure the technical decision. Referral traffic is a business metric; it is not an authentication mechanism, a capacity plan, or a reason to grant a bot unrestricted access to every route on a website.
A Windows-based site can expose much more than an article archive. Public IIS deployments frequently place documentation, help-center search, product catalogues, authenticated portals, API endpoints, file-download pages, staging paths, verbose error output, and legacy application routes behind the same hostnames or CDN rules. A broad crawler exception written to preserve AI visibility can become an unintended exception for expensive pages, poorly protected endpoints, or content the organization did not intend to make easy to enumerate.
The right first question is therefore not whether Meta “owes” a site a visit. It is whether the specific request is reaching only content that the operator has consciously chosen to expose to that specific verified service.
The visible user-agent is not proof of who connected
The more important security finding in DataDome’s earlier 2026 report is easy to miss amid the traffic totals. In January and February, DataDome said it observed 16.4 million spoofed requests claiming to be Meta-ExternalAgent, making that agent the most impersonated identity in its dataset. It also warned that relying on a familiar user-agent string as the basis for allowlisting can turn an access-control exception into an attack surface.
This is conventional bot security, not a new AI-specific vulnerability. An HTTP client can send almost any User-Agent header. A request labelled Meta-ExternalAgent, GPTBot, bingbot, or a major browser is only a claim until it is validated through the provider’s documented verification approach and the source network information associated with the request.
That should sharply limit the use of simple user-agent allowlists in IIS rewrite rules, Azure Front Door rules, Web Application Firewall policies, reverse proxies, rate-limit exemptions, and application logic. A rule that says “bypass rate limiting if the user-agent contains Meta” is not admitting Meta; it is admitting anyone able to type that string into a request header.
DataDome has an obvious commercial interest in emphasizing impersonation and centralized bot classification, so its spoofing counts should be treated as its own network observation rather than a global rate. The practical recommendation is sound regardless. Admins should verify automated traffic before applying exceptions, and they should retain enough request data to investigate claims of crawler activity later.
This also complicates the common advice to “just block the bot in robots.txt.” The Robots Exclusion Protocol is a cooperation signal for compliant crawlers; it is not a firewall. It does not stop a malicious scraper, a crawler using a forged identity, or a direct request to an endpoint. If a page must not be fetched, indexed, or exposed, access controls need to enforce that requirement at the origin, edge, identity, or application layer.
Google negotiations and Meta crawling are separate decisions
The submitted Search Engine Journal article argues that publishers are concentrating their leverage on Google because Google’s AI answers are changing a longstanding bargain: Google indexed pages, then sent users to those pages. The concern is real for sites dependent on search referrals, but it does not create a universal “allow AI” or “block AI” answer.
Microsoft operators should separate at least four questions that often get folded into one:
- A site may want conventional search indexing while declining model-training collection.
- A site may permit AI-answer indexing for public documentation while restricting high-cost dynamic pages.
- A site may accept verified crawlers but reject every traffic class that merely claims to be one.
- A site may prioritize uptime and egress cost even when its content is intentionally public.
The major mistake is assuming that a choice about Google’s public search presence automatically applies to Meta, OpenAI, Anthropic, Perplexity, Microsoft Bing, or an unknown agent using a browser-like signature. These services have different crawler identities, documented controls, verification methods, traffic patterns, and business purposes. An organization cannot govern them responsibly with a single rule called “AI bots.”
The licensing context reinforces the split between large publishers and ordinary operators. Meta’s agreement with News Corp, reported by The Wall Street Journal and confirmed by Engadget in March, was said to be worth up to $50 million per year and to cover content for Meta AI products. Meta had also announced commercial agreements with CNN, Fox News, USA Today, Le Monde, and other publishers.
Those deals are evidence that licensed content has value to Meta. They are not evidence that a midsize business, software vendor, community forum, or independent publisher has an equivalent negotiating position. Most site operators will not receive a licensing call. Their usable levers are technical: define accessible content, measure verified consumption, protect expensive routes, and decide what services receive an exception.
IIS and Azure teams should start with logs, not a blocklist
For a public site, the next step is an inventory rather than a dramatic robots.txt change. Pull 30 to 90 days of edge, WAF, and origin logs. Segment traffic by claimed user-agent, verified identity where available, source ASN or IP validation result, request rate, response status, bytes served, URI path, cache-hit ratio, and referrals that follow from the crawler or service.
On an IIS deployment, ensure the logging pipeline retains the cs(User-Agent) field and preserves the original client IP supplied by a trusted reverse proxy or Azure front door. If logs terminate at a CDN, do the first classification there; origin logs alone may show only proxy addresses. Application Insights, Azure Monitor, Microsoft Sentinel, and a SIEM can help correlate a crawler surge with backend CPU, bandwidth, cache misses, 429 responses, and abnormal traversal of search or download routes.
The useful output is a per-agent policy table maintained by the team that owns both the web service and its security controls. It should record the crawler identity, stated purpose, verification method, permitted paths, rate or concurrency limits, expiration date for the exception, and an accountable owner. That is mundane governance, but it prevents a temporary exception made for SEO from silently becoming a permanent security decision.
For sensitive routes, do not rely on bot labels at all. Require authentication, authorization, anti-automation controls, and sensible pagination or query limits. Public endpoints that trigger report generation, large downloads, expensive database searches, or AI-assisted operations should have their own limits regardless of whether the incoming client says it is a crawler, browser, or AI agent.
Allowlisting is an exception, not a trust model. If a known agent needs access, give it the narrowest validated access compatible with the intended outcome and monitor the resulting cost.
The story is a monitoring problem before it becomes a licensing fight
DataDome’s Q2 report says Meta’s two agents generated 9.1 billion requests across its network while producing almost no referral value. That conclusion is relevant to web operators because it demonstrates that crawl volume and business value can diverge sharply. A crawler that dominates dashboards may contribute nothing to demand generation, while a smaller agent could drive measurable user visits.
It does not follow that every Meta request is harmful or that every operator should block Meta-WebIndexer and Meta-ExternalAgent today. DataDome’s numbers do not establish the load, referral impact, or content value for any particular WindowsForum-style site, enterprise portal, or software documentation platform. Each operator has to measure its own service.
But the article’s central warning holds: the best-known AI company may not be the machine doing most of the reading. In an era when public web content is being consumed by training collectors, answer engines, agentic browsers, and forged bot identities, crawler management has become a production-operations and security discipline. The first practical milestone is simple: by the next reporting cycle, a site team should be able to identify its highest-volume automated clients, prove which ones are genuine, and explain exactly why each is allowed.