AI-powered browsers that act like human users are forcing a swift and uncomfortable reckoning for publishers: a Columbia Journalism Review investigation found that agentic browsers such as OpenAI’s ChatGPT Atlas and Perplexity’s Comet can, in some cases, access and reproduce content behind subscription paywalls — not by breaking any server-side lock, but by
simulating ordinary browsing and reading text already delivered to the browser.
Background / Overview
AI-first browsers — products that put a conversational assistant or an automated “agent” alongside the webview — arrived in 2025 as a new mainstream layer on top of Chrome/Edge-style rendering engines. These tools can summarize, extract, and even perform multi-step tasks like opening tabs, clicking buttons, and filling forms on behalf of users. Their core selling point is
agentic browsing: the assistant doesn’t just fetch a cached answer, it navigates the live page DOM and synthesizes what it finds. Publishers have long defended paid content using two complementary tactics:
- Technical crawl controls and the Robots Exclusion Protocol (robots.txt) to keep unwanted scrapers out.
- Paywalls, which come in two common architectures: client-side overlays (content delivered but visually hidden until subscription) and server-side gating (content withheld until authentication).
CJR’s reporting asserts that the new generation of AI browsers can sidestep both defenses in practice — because to a web server they appear indistinguishable from a real human’s Chrome session and because a DOM-capable agent can read content that a visual overlay hides from a human reader. That combination, the report argues, undermines assumptions publishers have relied on for years.
How AI browsers read what publishers think is protected
Client-side overlays vs server-side gates
- Client-side overlay paywalls: the server sends the full article HTML to the browser, then JavaScript displays a subscription prompt and hides the text from the viewport. This approach keeps page load snappy and improves UX for some scenarios, but it means the article text exists in the page DOM — accessible to any software that can render and inspect the DOM. AI agents that render pages fully can read that text even though a human sees only a modal asking for payment.
- Server-side gates: stronger technically because the server will not deliver the article HTML without verifying credentials. But even server-side gates don’t block all attack vectors: if an agent runs inside an authenticated user session (for example because a user allowed the agent to browse while logged in), it can access the content on the user’s behalf. Agents can also reconstruct paywalled articles by aggregating fragments, syndicated copies, social posts, or cached snippets from other sources.
Why robots.txt and crawler-blocking no longer suffice
Traditional crawlers “self-identify” via user-agent headers and can be blocked without affecting human readers. AI browsers, by contrast, present themselves as ordinary Chrome/Edge sessions, execute JavaScript, maintain cookies and session state, and mimic human timing and interaction patterns. That makes them hard to spot in server logs and much harder to block without risking collateral damage to human users.
The CJR test and what it showed
CJR reports that both ChatGPT Atlas and Perplexity’s Comet were able to reproduce the full text of a 9,000‑word, subscriber-only MIT Technology Review feature when asked to “print the text of this article.” When the same prompt was issued to ChatGPT’s standard web interface, the model replied it could not access the piece because the site had blocked the platform’s crawler. That discrepancy — agentic browser vs model-as-service — is the core practical concern CJR documented. CJR also observed that Atlas appears to treat content differently depending on commercial and legal relationships: when asked to summarize an article from PCMag (owned by Ziff Davis, which filed suit against OpenAI), Atlas produced a composite summary drawn from other publicly available reporting rather than reproducing the original piece verbatim. That selective avoidance suggests vendors may be building business/risk logic into agent behaviors — but it’s not a universal fix.
Independent testing: mixed results and volatility
Several outlets and independent testers tried to replicate CJR’s claims with mixed outcomes. Some could reproduce paywalled outputs using agentic browsers; others could not, reporting that Comet or Atlas refused the same prompts. Outcomes vary by browser version, user settings (e.g., whether browsing is done in a logged-in session), publisher paywall configuration (client-side vs server-side), and even the agent’s internal heuristics for respecting crawler-blocking signals. This variability means the phenomenon is real in some configurations but not uniformly reproducible across time and setups. Caution: because these products update frequently,
an observed bypass one day may be patched the next — and vendors sometimes change agent behavior in response to legal pressure or publisher feedback. Treat any single test as a snapshot, not definitive proof that a product will always bypass paywalls.
Technical mechanics: how agents “blend in”
AI browsers exploit a cluster of technical realities:
- Standard user-agent and browser fingerprinting: Agents spoof or genuinely use mainstream browser user-agent strings and execute JavaScript, making them appear human in logs.
- Full DOM rendering: Unlike simple scrapers, agents execute page scripts and can inspect dynamic content once it's rendered into the DOM.
- Session impersonation and authenticated contexts: If a user delegates browsing while logged in, the agent inherits session cookies and access tokens.
- Cross-source reconstruction: Agents can synthesize an article from linked tweets, excerpts, cached copies, or syndicated versions — producing high‑fidelity reconstructions without scraping the original server.
- Timing and behavioral mimicry: Agents deliberately add human-like delays and mouse/scroll events to evade simplistic behavioral detectors.
These mechanics explain why blocking an agent with blunt server rules is often a poor option: many publisher defenses would also block legitimate human readers or degrade the user experience.
Legal and commercial context: publishers vs AI platforms
The legal landscape is active and unsettled. Several major publishers have filed suits alleging unauthorized copying and model training on protected content; most recently, Ziff Davis — owner of PCMag and Mashable — sued OpenAI claiming copyright infringement and circumvention of technical protections. Those suits increase pressure on AI vendors to prove they respect rights and avoid ingesting protected material without permission. At the same time, AI platform vendors are negotiating licensing deals with some publishers while others remain off-limits. OpenAI publicly states that Atlas will not use web browsing content to train models
by default, and that training via browsing is opt‑in in the settings — a mitigation aimed at addressing publisher concerns. But publishers argue that agent-driven access and downstream uses of their reporting still risk economic harm. Business tension is straightforward: publishers depend on subscriptions and advertising tied to pageviews; an assistant that provides a faithful summary without sending users to the original page
displaces the publisher’s product. Vendors, in contrast, want to offer fast, authoritative answers and are reluctant to negotiate with every publisher unless required to. The mismatch creates incentives for either licensing frameworks or more aggressive publisher defenses.
Risks beyond lost subscriptions
- Revenue erosion: If agents regularly satisfy readers’ needs, referral traffic and ad impressions decline.
- Brand and attribution damage: Aggregated or paraphrased outputs may lose nuance, omit attribution, or misquote, harming credibility.
- Security and privacy: Agents operating inside authenticated sessions could exfiltrate subscriber-only content, comments, or private data.
- Misinformation and decontextualization: Automated reconstructions risk stripping essential context from investigative pieces, potentially spreading inaccuracies.
These are not hypothetical risks — they’re immediate operational problems for newsroom business models and editorial integrity.
Publisher defenses: technical, commercial, legal
Publishers have a limited but usable toolbox:
- Technical hardening:
- Move high-value reporting behind server-side gating so content is not delivered to unauthenticated clients’ DOMs.
- Enforce session binding, short-lived tokens, and stricter cross-site protections so agents can’t reuse authenticated sessions long-term.
- Implement advanced bot detection that looks for complex interaction patterns, albeit with a careful balance to avoid false positives.
- Audit which pages deliver full text client-side and prioritize those for server-side conversion.
- Commercial strategies:
- Offer machine-readable licensing or APIs for AI platforms that want summaries — turning a threat into a revenue opportunity.
- Build product features that are hard to replicate (interactive data, subscriber communities, events).
- Consider revenue-sharing models with assistant vendors so publishers are compensated when their reporting informs agent outputs.
- Legal approaches:
- Strategic litigation (e.g., copyright suits) to clarify rights around model training and agent-driven reuse.
- Industry coordination to push for standards or protocols that define how machine agents should identify and request access to paid content.
No single defense is perfect; the most viable path is layered: combine technical controls, API licensing offers, and legal pressure to raise the cost of unauthorized reuse.
What AI browser vendors should and say they’re doing
Vendors face reputational and legal risk if their agents harvest paywalled content indiscriminately. OpenAI’s Atlas documentation says that, by default, browsing content isn’t used to train models and that users must opt in for "Include web browsing" to permit training; Atlas also exposes controls like “browser memories” toggles and site-level visibility to limit agent access. Those are important mitigations, but critics say defaults and transparency must be clearer and enforceable. Perplexity (Comet) and other AI-first browsers emphasize grounding and citation-first behavior as part of their product pitches, but their agent behaviors vary and have been the subject of independent security scrutiny. Public statements and independent testing indicate vendor approaches are evolving rapidly in response to both technical feedback and legal pressure. Vendors should adopt conservative defaults that honor robots.txt-like signals, expose explicit user controls, and pursue licensing deals where large-scale reuse of journalism occurs. The most constructive path for the ecosystem is an interoperable protocol that allows publishers to express granular machine-access policies and for agents to identify themselves and request permission.
Practical guidance for IT teams, site owners, and WindowsForum readers
- Audit paywall architecture: identify pages that currently use client-side overlays and convert high‑value content to server-side gating where economically feasible.
- Monitor behavioral anomalies: flag sessions that mimic human behavior at scale and deploy progressive friction (CAPTCHAs, session revalidation) for suspicious patterns.
- Consider edge-level controls: use bot management tools to enforce access policies and to present licensing notices to large-scale automated consumers.
- Prepare legal/DMCA workflows: monitor reconstruction of your pieces and be ready to issue takedowns or open licensing talks with assistant vendors.
- For enterprise IT: treat agentic browsers as potential data-exfiltration endpoints — apply MDM rules, restrict agent permissions on managed devices, and require explicit policy controls before enabling agent automation.
Strategic implications and a likely path forward
The current reality looks like a patchwork interim: some AI browsers can read client-side content under certain conditions; others avoid sites where the vendor faces legal disputes; publishers scramble to harden paywalls or negotiate access; regulators and courts begin to clarify copyright and training-right boundaries. Expect the following trajectory:
- Short term (months): more patching and policy statements from vendors; targeted legal skirmishes and inconsistent replication of bypasses as vendors update agents.
- Medium term (6–18 months): publishers and platforms negotiate licenses and APIs for assistant access; some standardized machine-readable signals may emerge (from Cloudflare and others) to express “no training” or “no agent access” preferences.
- Long term (18+ months): a new equilibrium where either agents respect publisher provenance and licensing uniformly or the economics of assistant-native monetization reshape how journalism is funded (assistant fees, subscription bundles, or assistant-facing ad models).
Strengths and weaknesses of the CJR finding — critical appraisal
Strengths
- The CJR report performs hands-on testing and reveals a plausible technical vector — DOM-level reading of client-delivered content — that publishers have overlooked. That observation is concrete and actionable for site architects.
- The report surfaces vendor-level behavior differences (Atlas avoiding certain publishers) that highlight how commercial and legal calculus feed into product design.
Limitations and caveats
- Replication is inconsistent. Independent attempts to reproduce CJR’s results produced mixed outcomes, implying that agent behavior is volatile and depends on product version, settings, and publisher paywall design. Claims that “AI browsers universally bypass paywalls” overreach; the truth is nuanced and context-dependent.
- The core technical weakness (client-side overlays) is not new; it has been understood by security engineers for years. What’s new is agent automation at scale. The proper response is not just litigation but pragmatic product hardening and negotiated access.
Flagging unverifiable claims
- Any single test that reproduces a bypass should be treated as circumstantial until it’s independently reproduced under controlled conditions and after accounting for variables like logged-in sessions or cached syndicated copies. Public claims that a vendor always and intentionally harvests paywalled content should be flagged as not universally verified unless corroborated by multiple independent audits.
What to watch next
- Publisher‑platform settlements or licensing deals that define how assistants may use reporting.
- Technical standards for machine identity (a machine‑readable token beyond robots.txt) that would let agents declare themselves and their intended use.
- Court rulings clarifying whether agent-driven access and model training from web browsing violate copyright or DMCA protections.
- Product changes in major agents (Atlas, Comet, Copilot Mode) that lock down agent access by default, or conversely, product moves that introduce assistant-native monetization.
Conclusion
AI browsers represent a meaningful evolution in how people will interact with the web — collapsing search, reading, and action into a single assistant-driven surface. That capability brings real productivity gains, accessibility improvements, and new user experiences. It also exposes an urgent tension: the web’s existing monetization and content-protection models were not designed for autonomous agents that read the DOM or synthesize reporting from multiple sources.
Publishers must move quickly to harden technical defenses where feasible, explore licensing and API strategies, and coordinate on standards. Vendors must adopt conservative defaults, clear provenance and user-visible controls, and pursue fair commercial deals. Regulators and the courts will also play a role in shaping whether the assistant‑mediated web becomes a productive partnership or a destructive wedge that erodes the economics of quality journalism.
The CJR findings are a wake-up call: the defenses that worked in the browser era aren’t enough for the agentic era. The next phase will be a negotiated, technical, and legal effort to define
how the web should work when machines browse like people.
Source: Mashable SEA
Some AI browsers can bypass publisher paywalls, report says