Tech platforms and AI labs are operating on two different rulebooks: the same companies that ban automated scraping of their services in their terms of service are also building the next generation of generative models on training pipelines that — evidence shows — lean heavily on content harvested at scale from the public web, including copyrighted music, videos, and other creator work. The contradiction is the headline: industry players insist on permission and written consent for anyone using their platforms, while large-scale data collection practices to train AI proceed with little public oversight. This tension is at the center of a wave of reporting and a dossier compiled by the music‑publishing trade body ICMP — the very material summarized in the piece the user provided for review.
The last 18 months have seen high‑profile investigations and lawsuits probing how generative AI models are trained. One strand of the reporting examines music: a two‑year dossier assembled by the International Confederation of Music Publishers (ICMP) and shared with journalists alleges that major firms trained models on copyrighted songs and lyrics at scale, calling the practice “the largest intellectual property theft in human history.” The ICMP dossier claims catalog‑level lists, private dataset manifests, and analyses of model outputs that suggest replication of protected lyrics and musical structures.
A parallel investigative thread has documented massive video and transcript collections pulled from YouTube and re‑used as training corpora. Independent reporting has traced widely circulated datasets — for example, a YouTube‑subtitles collection and a host of multimillion‑clip video datasets used by academic and industry labs — back to automated scraping or third‑party aggregators. These sets have been tied, via published papers and leaked manifests, to model training at leading companies. Wired/Proof’s reporting and follow‑ups in outlets like The Verge establish that many datasets used in research and product work contain thousands to millions of YouTube IDs and transcripts. (wired.com)
At the same time, platform terms of service and developer policies tend to forbid exactly this sort of automated collection without prior written permission. YouTube’s developer policy explicitly bars scraping its Applications and requires prior written permission for automated collection beyond what robots.txt permits, while Facebook/Meta’s rules forbid automated collection without permission. That mismatch — “do as we say, not as we (or others) do” — is what commentators and rights holders now call a double standard. (developers.google.com)
But reporting and dataset leaks complicate these responses: models have been shown in litigation to reproduce lyrics or other copyrighted text, and investigative datasets have been tied to academic and corporate work. Moreover, companies often host internal datasets or rely on third‑party aggregators, introducing opportunities for unlicensed content to propagate in opaque ways. (theverge.com)
That reality creates both moral and commercial debt. Platforms and AI firms must reconcile their enforcement of Terms of Service with their data‑sourcing practices; creators and publishers must demand transparent, scalable licensing and provenance tools; and regulators should set clear standards for traceability and rights‑respecting use. Without these reforms, the double standard will persist: platforms will block others from scraping, while models trained on scraped content operate in plain sight — an untenable state for a healthy creator economy and for durable public trust in AI.
Short of an immediate global legal consensus, practical governance will emerge from four things: stronger provenance and logging standards, scalable licensing markets for training data, consistent ToS enforcement by platforms, and legal rulings that narrow the current grey areas around fair use for model training. Those are not trivial tasks, but the alternatives — lost revenues for creators, fractured legal battles, and mounting public mistrust — are far costlier. The datasets and dossiers that sparked this debate provide the technical breadcrumbs necessary to build a new bargain between creators and technologists; the challenge now is whether markets, platforms, and laws will move to meet that moment. (theverge.com)
Source: the-decoder.com Tech's data double standard: scrape to train, block everyone else
Background / Overview
The last 18 months have seen high‑profile investigations and lawsuits probing how generative AI models are trained. One strand of the reporting examines music: a two‑year dossier assembled by the International Confederation of Music Publishers (ICMP) and shared with journalists alleges that major firms trained models on copyrighted songs and lyrics at scale, calling the practice “the largest intellectual property theft in human history.” The ICMP dossier claims catalog‑level lists, private dataset manifests, and analyses of model outputs that suggest replication of protected lyrics and musical structures.A parallel investigative thread has documented massive video and transcript collections pulled from YouTube and re‑used as training corpora. Independent reporting has traced widely circulated datasets — for example, a YouTube‑subtitles collection and a host of multimillion‑clip video datasets used by academic and industry labs — back to automated scraping or third‑party aggregators. These sets have been tied, via published papers and leaked manifests, to model training at leading companies. Wired/Proof’s reporting and follow‑ups in outlets like The Verge establish that many datasets used in research and product work contain thousands to millions of YouTube IDs and transcripts. (wired.com)
At the same time, platform terms of service and developer policies tend to forbid exactly this sort of automated collection without prior written permission. YouTube’s developer policy explicitly bars scraping its Applications and requires prior written permission for automated collection beyond what robots.txt permits, while Facebook/Meta’s rules forbid automated collection without permission. That mismatch — “do as we say, not as we (or others) do” — is what commentators and rights holders now call a double standard. (developers.google.com)
What the dossiers and investigations actually show
ICMP’s music dossier — scale, specificity, and claims
The ICMP claims it spent two years compiling evidence that AI developers have ingested copyrighted music and lyric datasets from YouTube, public repositories, leaks, and private manifests. Their public statements — reinforced in the reporting summarized by the provided article — assert:- Manifest lists containing URLs that map to platform content.
- Private datasets tied to startups and research projects.
- Model output samples and court filings that, they say, show memorization or replication of copyrighted lyrics.
- An estimate that training sets sometimes contain “tens of millions” of musical works and that infringement is ongoing at scale.
YouTube and video datasets — the hard numbers
Independent investigations have documented specific datasets built from YouTube subtitles and clips. Proof News (copublished with Wired) analyzed a public dataset of YouTube subtitles containing roughly 173,536 video entries from over 48,000 channels and showed how that data had been included in widely reused corpora such as “the Pile.” Other datasets used in academic and industrial research reach into the millions or tens of millions of clips — HowTo100M, ACAV100M, HD‑VILA, and similar efforts are explicitly curated from YouTube IDs and sometimes reuse automated captions or scraped metadata. Reporting also surfaced spreadsheets and leaked manifests linked to commercial projects (for example, a Runway leak indicating prioritized video channels), suggesting selection criteria and manual curation were involved, not accidental inclusion. (wired.com)Legal and courtroom evidence
Music‑industry lawsuits have already produced court filings and settlements that bear on the question of data use. Anthropic’s litigation over song lyrics produced filings where plaintiffs cited model outputs and alleged ingestion; Anthropic agreed to certain “guardrails” around reproducing lyrics as part of litigation compromises. More broadly, an increasing body of litigation — covering news publishers, book authors, visual artists, and music publishers — is forcing discovery orders that could reveal ingestion practices and training manifests. Reuters and other outlets have tracked these legal developments and the partial settlements or judge‑ordered interventions that followed. (reuters.com)Why the “double standard” matters — practical and legal dimensions
1. Contractual and policy inconsistency
Platforms publish terms that prohibit automated collection without permission but host the very content later found in training datasets. This matters for two reasons: creators often rely on platform terms to control reuse (and monetization) of their work; and companies that enforce ToS selectively may be seen as privileging their commercial AI ambitions over contract enforcement. The relevant clauses are explicit: YouTube’s developer policies disallow scraping beyond robots.txt or prior permission, and Meta’s developer terms forbid automated collection unless authorized. (developers.google.com)2. Market and competitive fairness
If AI companies can ingest creators’ work to train competing generative models without paying or licensing, they gain a near‑free path to replicate creative labor at scale. Rightsholders argue that this undermines existing licensing markets and reduces economic incentives for creators and publishers. ICMP’s rhetoric — which frames the practice as massive IP theft — underscores the industry’s existential worry: generative systems trained on unpaid creative labor could displace human writers, songwriters, and producers while capturing surplus value for the platform or model owner.3. Technical traceability and regulatory promises
A common industry response has been that “disclosing training sources” is complicated. The dossiers and dataset manifests suggest the opposite: scraped content is often meticulously labeled with metadata (artist, genre, tempo, IDs) and assembled in reproducible lists. That level of structure means traceability is technically feasible — which is central to proposals in the EU AI Act and other regulatory frameworks that would require provenance logs and recordkeeping for data used to train high‑risk models. In short, the “too hard to trace” defense looks weaker when dataset manifests and leaked spreadsheets exist. (acav100m.github.io)Notable strengths and weaknesses of the evidence
Strengths
- Multiple independent investigations converge on similar patterns: subtitle and video datasets include creator content compiled from YouTube; leaked manifests and spreadsheets exist; and court filings show reproductions of copyrighted lyrics in model outputs. This multiplicity strengthens the inference that ingestion is not isolated or accidental. (wired.com)
- Some datasets are public, or semi‑public, enabling reproducible checks (e.g., proofs that specific video IDs appear in training lists). That makes allegations verifiable in principle, reducing the “he said / she said” aspect of the debate. (proofnews.org)
- Legal discovery and settlements have already compelled companies to adopt guardrails, showing the claims have operational and judicial traction. (reuters.com)
Weaknesses and caveats
- Presence in a manifest is not a smoking gun of deliberate ingestion into a production model: training pipelines are heterogeneous, and datasets are often sampled, filtered, or partially used. Plaintiffs must usually prove not just the dataset’s existence but that the defendant used those exact records to train a commercial model. Courts require precise discovery to bridge that gap.
- Some items labeled “evidence” are weaker: conversational chatbot “admissions” about training data can be misleading because LLMs are probabilistic and can hallucinate or generalize. Those chatbot statements are low‑probative unless backed by logs or manifests.
- Industry defense of “fair use” for training remains legally unresolved in the U.S.; outcomes vary by jurisdiction and by the facts shown in discovery. Even when courts find training can be fair use in theory, storage of pirated copies or near‑verbatim reproduction can tip the scales back toward infringement. Recent rulings and settlement frameworks reflect this nuance. (investing.com)
The companies’ responses and the public record
Tech firms have offered a spectrum of answers: public denials or guarded statements emphasizing use of “publicly available” or licensed datasets, claims of third‑party data sourcing, and promises of technical guardrails to prevent verbatim reproduction of protected lyrics or videos. Some have pointed to research projects trained on public or licensed corpora (for example, OpenAI’s historical Jukebox research disclosure of 1.2M songs for a research model), while others emphasize content filters or output controls.But reporting and dataset leaks complicate these responses: models have been shown in litigation to reproduce lyrics or other copyrighted text, and investigative datasets have been tied to academic and corporate work. Moreover, companies often host internal datasets or rely on third‑party aggregators, introducing opportunities for unlicensed content to propagate in opaque ways. (theverge.com)
Practical risks and systemic harms
- Creator displacement and economic harm. If models can cheaply generate convincing substitutes for songs, news articles, or video formats, revenue streams for creators — licensing, performance, and derivative works — may shrink. The music industry’s loud reaction stems from an existential betting that unlicensed model training can undercut music’s value chain.
- Regulatory blowback and compliance costs. Companies that continue opaque ingestion may face fines, injunctive relief, or obligations under laws like the EU AI Act. Those outcomes could force expensive remediation, model retraining, or royalties. The legal landscape is shifting quickly; the costs of noncompliance could be material. (reuters.com)
- Reputational risk and creator pushback. High‑profile creators and large publishing houses can marshal public opinion and legal resources. Platform trust erodes when creators feel exploited; platforms rely on creators for content and engagement. Recent opt‑in controls (YouTube allowing creators to opt into third‑party AI training) are partly a reaction to that pressure. (theverge.com)
- Technical debt: poisoned or misattributed datasets. Datasets assembled from scraped web content often include watermark‑stripped or rehosted materials, raising provenance and contamination risks. If models are trained on mislabeled data, compliance audits and provenance tracking become expensive but essential. (acav100m.github.io)
What the industry, regulators, and creators should do — pragmatic steps
For platforms and AI firms
- Implement mandatory provenance logging. Record per‑work provenance for datasets used in training and maintain those records for a fixed audit horizon (for example, 7–10 years). This is feasible: leaked manifests show metadata is commonly recorded. Provenance reduces legal uncertainty and supports transparency.
- Adopt creator opt‑in / licensing mechanisms at scale. Platforms can operationalize creator opt‑ins (as YouTube began rolling out) and streamline licensing marketplaces to enable lawful dataset access and revenue sharing.
- Enforce platform TOS consistently. If scraping is prohibited by platform terms, enforcement must be coherent. Selective or inconsistent enforcement invites litigation and policy pushback.
- Offer contractual non‑use and enterprise controls. Vendors should provide enterprise customers contractual guarantees about non‑use and data segregation, including clear opt‑out mechanisms for organizations with regulatory requirements.
For creators and publishers
- Use platform controls and metadata tools to indicate reuse preferences.
- Pursue collective licensing schemes that scale to AI training scenarios and provide predictable remuneration.
- Invest in watermarking and other provenance technologies that make unauthorized reuse easier to detect.
For regulators and policymakers
- Require data provenance obligations for high‑impact AI models so that rights holders can identify training sources.
- Clarify the scope of fair use in training contexts or establish a licensing default to prevent legal fragmentation.
- Support technical standards for watermarking, dataset manifests, and audit trails to reduce disputes and facilitate compliance.
Where the evidence remains uncertain — cautionary notes
- Not every dataset manifest or leaked spreadsheet equates to proven infringement. Demonstrating legal liability typically requires proving that an identified defendant ingested and used the specific content in a commercial model and that the use was not a protected fair use. Discovery will be decisive here, and courts remain split on aspects of the law.
- Chatbot “admissions” about training data are inherently noisy. Models can hallucinate, so a model’s conversational claim that “I was trained on X” is not prima facie evidence. Forensic datasets, training manifests, and logs are the gold standard; those are often sealed behind litigation and corporate confidentiality.
- Some companies have begun lawful, opt‑in licensing deals and guardrail investments; wholesale condemnation without recognizing those efforts risks oversimplifying a diverse landscape.
Conclusion — a practical verdict for the era of generative AI
We are at a regulatory and ethical inflection point. The evidence marshaled by rights holders and investigators — dataset manifests, investigative reporting, leaked spreadsheets, and litigation exhibits — paints a coherent picture: large pools of creator work have been harvested and repurposed for model training, sometimes in ways that appear to conflict with platform policies and the expectations of creators. (wired.com)That reality creates both moral and commercial debt. Platforms and AI firms must reconcile their enforcement of Terms of Service with their data‑sourcing practices; creators and publishers must demand transparent, scalable licensing and provenance tools; and regulators should set clear standards for traceability and rights‑respecting use. Without these reforms, the double standard will persist: platforms will block others from scraping, while models trained on scraped content operate in plain sight — an untenable state for a healthy creator economy and for durable public trust in AI.
Short of an immediate global legal consensus, practical governance will emerge from four things: stronger provenance and logging standards, scalable licensing markets for training data, consistent ToS enforcement by platforms, and legal rulings that narrow the current grey areas around fair use for model training. Those are not trivial tasks, but the alternatives — lost revenues for creators, fractured legal battles, and mounting public mistrust — are far costlier. The datasets and dossiers that sparked this debate provide the technical breadcrumbs necessary to build a new bargain between creators and technologists; the challenge now is whether markets, platforms, and laws will move to meet that moment. (theverge.com)
Source: the-decoder.com Tech's data double standard: scrape to train, block everyone else
