Common Crawl
From AltData.wiki, The Alternative Data Encyclopedia
Common Crawl is a nonprofit 501(c)(3) organization that crawls the web and freely provides its archives and datasets to the public. Founded in 2007 by Gil Elbaz, it maintains an open repository of web crawl data comprising more than 300 billion pages spanning 15 years, with 3-5 billion new pages added each month.[1][2][3].
The corpus is hosted primarily on Amazon Web Services, where access to the data is free although users may incur storage and compute costs.[2] It has been cited in more than 10,000 research papers and became a foundational training source for large language models in the 2020s.[1][2]
What It Does
Common Crawl performs wholesale crawls of the open web and publishes the resulting raw HTML, metadata extracts, and derived indexes so that researchers, startups, and analysts can analyze web-scale data without operating their own crawlers.[1]
Its crawler, known as CCBot, respects robots.txt exclusions, and the foundation maintains a public opt-out registry documenting removal requests.[1]
Products
The organization publishes monthly crawl archives alongside several derived data products documented on its site: the CDXJ index and URL index for looking up captures, and Web Graphs built from hyperlink structure, plus public crawl and graph statistics.[1]
In April 2026 it experimentally began distributing crawl archives through a Hugging Face storage bucket in addition to its standard Amazon S3 hosting.[2]
Data And Methodology
The project was designed to offer fresher, more analyzable data than the Internet Archive's Wayback Machine, and to reduce dependence on closed search-engine indexes. Early crawls started from a list of URLs donated by the search engine Blekko.[2]
Because the corpus is raw web text, downstream organizations typically refine it into filtered datasets; notable derivatives include Google's C4 (built in 2019 to train the T5 models) as well as FineWeb and DCLM.[2]
Buyers And Use Cases
There is no commercial product; use cases are defined by the research community. Documented applications include language-model training corpora, language identification benchmarks, economic studies that track AI adoption across firms via their websites, geolocated news-article datasets, censorship research, and studies of crawler blocking behavior.[1]
By the early 2020s AI companies had become major consumers: ChatGPT was reported in 2023 as largely trained on Common Crawl data, and the foundation received donations from AI firms including Anthropic and OpenAI.[2]
History
Common Crawl was founded in 2007 in San Francisco and began archiving websites that year, publishing its first crawls in 2011. By 2015 the corpus held 1.8 billion webpages, and AWS support kept computing costs to an average of $2,000-4,000 per month at that time.[2]
Until 2023 it was funded almost exclusively by the Elbaz Family Foundation Trust, after which donations from the AI industry became significant. Rich Skrenta serves as executive director.[2]
Compliance And Controversy
The foundation's relationship with publishers has been contested. In November 2025, an investigation by *The Atlantic* reported that Common Crawl misled publishers about respecting paywalls and about honoring content-removal requests, showing no entries in its public search for sites that had opted out while still including them in scrapes used by AI companies; CCBot was found to be the most widely blocked bot among the top 1,000 websites.[2]
A 2024 study cited by Wikipedia found that 45% of C4's content was restricted by websites' terms of service for uses such as for-profit AI training, illustrating the legal ambiguity surrounding web-scale corpora.[2]