Google and Bing have quietly but decisively told publishers: stop creating separate markdown or plain‑text pages specifically for large language models — it’s unnecessary, risky, and likely to backfire for sites that chase short-term AI visibility gains. Search teams at both companies say their crawlers and AI systems already understand modern HTML and semantic markup; duplicating your site into a machine‑only layer (the so‑called “markdown‑for‑LLMs” approach) introduces duplicate content, maintenance headaches, and potential policy friction without delivering meaningful advantage.
The last 18 months have seen a flurry of creative — and sometimes desperate — experiments aimed at courting AI‑driven discovery. Publishers, CMS vendors, and SEO practitioners noticed that AI assistants and “answer engines” increasingly synthesize content rather than simply listing links. In response, a grassroots proposal called llms.txt (and related ideas like hosting markdown mirrors or llms‑full.txt bundles) gained traction: publish a concise, machine‑friendly manifest at /llms.txt and optionally add a .md mirror for every important HTML page so LLMs could ingest a clean, stripped‑down version. Proponents framed it as the next robots.txt or sitemap.xml — a way to make the web overtly agent‑friendly.
That proposal triggered two waves of adoption: tooling vendors added llms.txt generators and CMS plugins (Yoast, Drupal modules, custom generators), and site teams experimented with automated markdown mirrors for docs, product pages, and knowledge bases. Some early adopters reported anecdotal improvements in how third‑party chatbots summarized their content — enough to convince more teams to test the pattern. But the major search platforms didn’t sign on.
Key, high‑impact steps that actually move the needle for both traditional search and AI answers:
Publishers worry that AI summaries reduce referral traffic and undermine the advertising or subscription models that fund journalism and specialist content. In some cases publishers have negotiated licensing deals with AI platforms as a defensive response. Industry incidents — like the public controversy when documentation‑centric businesses reported steep traffic declines after the rise of answer engines — underscore the economic stakes.
At the same time, the marketplace of standards is evolving: community proposals (llms.txt, llms‑full.txt), vendor‑specific opt‑outs, and nascent specs for content signals, TDM (text‑and‑data‑mining) reservations, and agent manifests are all in play. None of these is a universal standard yet — major AI operators have not committed to llms.txt as a protocol, and server logs show limited crawler interest across the board. That reality helps explain Google and Bing’s reluctance to endorse an ecosystem that could fragment the web and make indexing more brittle.
Strengths of the search engines’ guidance:
Source: WebProNews Google and Bing Reject the Markdown-for-LLMs Trend: Why Search Engines Say Stop Building Separate Pages for AI Crawlers
Background: the markdown‑for‑LLMs fad and how it rose fast
The last 18 months have seen a flurry of creative — and sometimes desperate — experiments aimed at courting AI‑driven discovery. Publishers, CMS vendors, and SEO practitioners noticed that AI assistants and “answer engines” increasingly synthesize content rather than simply listing links. In response, a grassroots proposal called llms.txt (and related ideas like hosting markdown mirrors or llms‑full.txt bundles) gained traction: publish a concise, machine‑friendly manifest at /llms.txt and optionally add a .md mirror for every important HTML page so LLMs could ingest a clean, stripped‑down version. Proponents framed it as the next robots.txt or sitemap.xml — a way to make the web overtly agent‑friendly. That proposal triggered two waves of adoption: tooling vendors added llms.txt generators and CMS plugins (Yoast, Drupal modules, custom generators), and site teams experimented with automated markdown mirrors for docs, product pages, and knowledge bases. Some early adopters reported anecdotal improvements in how third‑party chatbots summarized their content — enough to convince more teams to test the pattern. But the major search platforms didn’t sign on.
Where Google and Bing stand now
Leading voices at Google and Microsoft have publicly discouraged building separate LLM‑only pages.- Google’s John Mueller has repeatedly said major AI services haven’t adopted llms.txt and that converting live pages to markdown solely for LLM consumption is unnecessary — and in his words, “less is more in SEO.” That view is echoed in Search Console guidance emphasizing that there are no extra special requirements to appear in AI Overviews beyond good HTML and structured data.
- On the Bing side, Fabrice Canel (Principal Program Manager for Bing’s crawling infrastructure) warned that serving separate bot‑only content doubles crawl load, creates maintenance and similarity checks for crawlers, and frequently results in broken or neglected bot‑facing pages. Bing’s team encourages publishers to use schema and better HTML structure rather than maintain parallel content streams.
Why the idea seemed attractive — and why it spread
The appeal of llms.txt and markdown mirrors was simple.- LLMs historically have context window limits and work better with concise, factual inputs. A neat markdown file stripped of ads, navigation, and scripts looks like a clean prompt.
- Publishers were watching referral traffic shift as AI assistants produced fully formed answers and worried that zero‑click answers would cannibalize visits.
- The effort to publish a /llms.txt or to auto‑generate .md versions seemed low cost, relative to the perceived upside of being cited directly by an assistant.
The technical reality: duplication, maintenance, and the crawl paradox
Putting aside policy and optics for a moment, the markdown‑mirror approach creates genuine technical problems.- Duplicate content risk. Hosting markdown versions of the same content at different URLs expands your site’s footprint and can dilute ranking and citation signals. Canonical tags can reduce harm but they don’t eliminate crawler overhead or the potential for indexing errors. Search engines have long treated near‑duplicate pages as lower‑value, and adding a machine‑focused copy effectively doubles that risk.
- Content drift and operational burden. Every content update now requires synchronization across two surfaces. For large documentation sites, product hubs, or news outlets with frequent edits, keeping the markdown mirrors consistent is a nontrivial engineering effort. When the two versions fall out of sync, you risk outdated or contradictory content being used by AI systems — which harms credibility and may lead to citation mismatches.
- Crawling inefficiency and verification. Search platforms will still crawl the canonical HTML to verify that human‑facing content matches any bot‑facing version. That eliminates the supposed bandwidth advantage of a small markdown manifest and imposes extra checks (and potential throttling) on your site. Bing’s team has explicitly called this out as counterproductive.
- Cloaking and policy exposure. Serving different content to crawlers than to users has always been a red flag in search best practices. Even when the intent is benign (optimizing for AI understanding), the pattern resembles cloaking, which can expose a site to ranking penalties or manual review. Industry experts have raised that exact concern in public debates.
What search engines actually need: good HTML and clear signals
The unanimous recommendation from search teams is straightforward: stop trying to feed models a secret diet. Instead, make your human‑facing HTML as transparent, structured, and usable as possible.Key, high‑impact steps that actually move the needle for both traditional search and AI answers:
- Use semantic headings (H1–H6) and proper HTML5 structure so parsers can easily find the main content area.
- Mark up entities and content types with structured data (JSON‑LD / Schema.org) — Article, FAQPage, Product, HowTo — to tell systems what a page ust author bylines, dates, and provenance information so AI systems have natural signals for authority and recency.
- Optimize performance and accessibility: fast, well‑structured pages are easier to crawl and more likely to be selected as a grounding source.
- Publish machine‑readable sitemaps and keep robots.txt accurate; use existing opt‑out mechanisms (robots, vendor‑specific flags) if you deliberately want to block some crawlers from training on your content.
The broader debate: who controls how AI consumes the web?
The markdown‑for‑LLMs conversation sits at a larger intersection of economics, copyright, and web governance.Publishers worry that AI summaries reduce referral traffic and undermine the advertising or subscription models that fund journalism and specialist content. In some cases publishers have negotiated licensing deals with AI platforms as a defensive response. Industry incidents — like the public controversy when documentation‑centric businesses reported steep traffic declines after the rise of answer engines — underscore the economic stakes.
At the same time, the marketplace of standards is evolving: community proposals (llms.txt, llms‑full.txt), vendor‑specific opt‑outs, and nascent specs for content signals, TDM (text‑and‑data‑mining) reservations, and agent manifests are all in play. None of these is a universal standard yet — major AI operators have not committed to llms.txt as a protocol, and server logs show limited crawler interest across the board. That reality helps explain Google and Bing’s reluctance to endorse an ecosystem that could fragment the web and make indexing more brittle.
Practical checklist for publishers: what to do today (and what not to do)
If you manage a news site, documentation hub, ecommerce catalog, or developer portal, here’s a prioritized action list that respects both search engine guidance and the realities of AI discovery.- Prioritize the canonical HTML
- Use clear H1/H2 structure and place the main content in semantic containers (article, main).
- Ensure your CMS renders accessible, server‑side HTML so crawlers don’t need complex JS execution to get core content.
- Add authoritative structured data
- Implement JSON‑LD for Article, Product, FAQPage, HowTo, Organization, and Person where relevant.
- Include publish dates, author names, and publisher metadata.
- Improve provenance and trust signals
- Add author bios, editorial policies, and clear bylines.
- Publish attribution and source lists for research or reporting pieces.
- Harden technical basics
- Submit sitemaps and monitor indexing in Search Console and Bing Webmaster Tools.
- Keep robots.txt accurate; use vendor‑specific training/extended flags where supported (e.g., Google’s training flags if applicable).
- Monitor AI surfaces
- Track changes in referral behavior and impressions from AI Overviews, Copilot, and other assistant surfaces. Don’t treat zero‑click decline as immediate content failure — diagnose query types and intent shifts first.
- Resist the duplicate‑content shortcut
- Don’t auto‑generate .md mirrors of every page just for AI crawlers. If you do experiment with an llms.txt manifest, treat it as a proactive note to future agents, not as a substitute for real page quality. Keep any llms.txt small, focused, and synchronized with canonical pages.
- If you absolutely must publish an /llms.txt
- Keep it minimalist: a short summary and a curated list of canonical pages.
- Avoid listing behind‑login or gated content unless you intend it for public retrieval.
- Treat it as a low‑cost experiment, not a strategic dependency.
The future: standards, content signals, and emergent protocols
The web is likely to converge on a small set of interoperable signals for agent discovery and rights management — but the timeline and shape of those standards remain uncertain.- Community proposals and vendor experiments are exploring manifests, Content Signals, TDM reservation headers, and Agent Manifests (WebMCP / MCP-type manifests). These are promising because they aim to solve two problems at once: discoverability for legitimate retrieval, and governance for training/licensing preferences. But adoption depends on AI platforms and browser vendors agreeing on conventions.
- The sist as a proactive, low‑cost shim while standards mature: many organizations publish them as future‑proofing. Yet major AI operators have been clear they haven’t standardized on the file and don’t depend on it today. That means llms.txt is a potential tool, not a replacement for good HTML and schema today.
- Watch the space for two converging trends: (1) emerging HTTP headers and /.well‑known manifests offering machine‑readable licensing and update frequency signals; and (2) search engines baking provenance and evidence selection into answer generation so that citations and click‑through incentives become more transparent. Publishers should follow these discussions and be ready to adapt, but not to dismantle their existing publishing pipelines for speculative short‑term gains.
Critical analysis: strengths, risks, and blind spots in the “don’t‑mirror” advice
There’s strong merit in the stance taken by Google and Bing: the web must remain a single, authoritative layer where humans and machines read the same content. This preserves simplicity, reduces duplication, and prevents a race to game transient model behaviors.Strengths of the search engines’ guidance:
- It preserves long‑standing core SEO principles that are broadly understood and operationalized across publishing stacks.
- It avoids creating incentives to hide or split content between audiences (human vs. machine), a pattern that historically invited abuse.
- It reduces the operational complexity for webmasters who would otherwise have to support parallel publishing workflows.
- Economic realities for publishers are real: zero‑click AI consumption can materially reduce referral traffic and the ad/subscription revenue that supports journalism and deep technical docs. Saying “fix your HTML” does not directly solve the business model challenge of AI‑mediated answers.
- The absence of a standardized, enforceable mechanism for distinguishing retrieval vs. training rights leaves publishers little recourse when they object to how their content is used. Until a widely adopted, machine‑enforceable protocol exists, publishers will continue to experiment with ad hoc solutions (including llms.txt variants).
- Not all publishers have engineering bandwidth to implement perfect semantic HTML and structured data at scale. Tools that automate production of high‑quality structured markup are still catching up to demand.
A short, conservative playbook to protect discovery and revenue
- Audit your traffic: identify pages with the biggest AI‑era referral declines and prioritize them for structured‑data and provenance fixes.
- Strengthen “proof” on pages: add author credentials, editorial notes, citations, and primary data where possible.
- Invest in gated or productized hooks: tutorials, interactive tools, or gated add‑ons that aren’t trivially replaced by a generated snippet.
- Monitor and document: keep logs, server evidence, and analytics to trace AI‑surface impacts — these will be essential for any future negotiations or enforcement cases.
- Participate in standards work: join or follow WebMCP, Content Signals, and relevant W3C community groups so you can influence interoperable solutions rather than react to them.
Conclusion
The short answer to the markdown‑for‑LLMs fad is blunt and useful: don’t do it as a core strategy. Google and Bing have explained why — their crawlers already parse and prioritize well‑structured HTML, and maintaining a parallel markdown layer creates duplication, maintenance cost, and potential policy problems without proven benefit. Instead, publishers should double down on durable investments that improve both human and machine understanding: semantic HTML, JSON‑LD structured data, clear provenance, and performance‑first publishing. That approach aligns with how search engines select sources for both traditional results and generative answers, and it preserves a single, auditable canonical web layer that benefits users, developers, and machines alike.Source: WebProNews Google and Bing Reject the Markdown-for-LLMs Trend: Why Search Engines Say Stop Building Separate Pages for AI Crawlers
