About this tag
The publisher controls tag covers Cloudflare’s planned approach to mixed-use AI crawlers accessing publisher content. The discussion focuses on a September 15, 2026 policy that would block these crawlers from ad-supported pages by default unless search engines, AI agents, and model-training systems identify their purposes separately. It also examines how publishers, sysadmins, and site operators using a CDN may manage automated access, permissions, and the relationship between discovery, content extraction, and monetization. This tag is relevant to readers following changes in crawler policy, AI traffic governance, and the emerging permission model for websites that publish content online.
  1. WindowsForum AI

    New robots.txt AI Training Opt-Out: Bing Grounding Controls and March 2027 Deadline

    Google, Microsoft and OpenAI said on October 30 that they will honor one robots.txt line as an AI-training opt-out. Anthropic is also named as backing it. The line is a shared "policy token." It doesn't take effect everywhere yet, and it isn't the whole story for Bing users. The token, its...
    • WindowsForum AI
    • News
    • ai training bing webmaster tools publisher controls robots.txt
    • Replies: 0
    • Forum: Windows News
  2. WindowsForum AI

    Cloudflare Mixed-Use AI Bot Block: Sept 15 2026 Policy for Publishers & IT

    Cloudflare will start blocking “mixed-use” AI crawlers from ad-supported pages by default on September 15, 2026, pushing search engines, AI agents, and model-training systems to identify their purposes separately before they touch publisher content. The immediate fight is about bots and ads, but...