Common Crawl
Free & Open alternative data provider · 1 listed dataset
Datasets from Common Crawl (1)
Common Crawl is a nonprofit 501(c)(3) organization that crawls the web and freely provides its archives and datasets to the public. Founded in 2007 by Gil Elbaz, it maintains an open repository of web crawl data comprising more than 300 billion pages spanning 15 years, with 3-5 billion new pages added each month.[1][2]
The corpus is hosted primarily on Amazon Web Services, where access to the data is free although users may incur storage and compute costs.[2] It has been cited in more than 10,000 research papers and became a foundational training source for large language models in the 2020s.[1][2]
What It Does
Common Crawl performs wholesale crawls of the open web and publishes the resulting raw HTML, metadata extracts, and derived indexes so that researchers, startups, and analysts can analyze web-scale data without operating their own crawlers.[1]
Its crawler, known as CCBot, respects robots.txt exclusions, and the foundation maintains a public opt-out registry documenting removal requests.[1]
Products
The organization publishes monthly crawl archives alongside several derived data products documented on its site: the CDXJ index and URL index for looking up captures, and Web Graphs built from hyperlink structure, plus public crawl and graph statistics.[1]
In April 2026 it experimentally began distributing crawl archives through a Hugging Face storage bucket in addition to its standard Amazon S3 hosting.[2]
Data And Methodology
The project was designed to offer fresher, more analyzable data than the Internet Archive's Wayback Machine, and to reduce dependence on closed search-engine indexes. Early crawls started from a list of URLs donated by the search engine Blekko.[2]
Because the corpus is raw web text, downstream organizations typically refine it into filtered datasets; notable derivatives include Google's C4 (built in 2019 to train the T5 models) as well as FineWeb and DCLM.[2]
Buyers And Use Cases
There is no commercial product; use cases are defined by the research community. Documented applications include language-model training corpora, language identification benchmarks, economic studies that track AI adoption across firms via their websites, geolocated news-article datasets, censorship research, and studies of crawler blocking behavior.[1]
By the early 2020s AI companies had become major consumers: ChatGPT was reported in 2023 as largely trained on Common Crawl data, and the foundation received donations from AI firms including Anthropic and OpenAI.[2]
History
Common Crawl was founded in 2007 in San Francisco and began archiving websites that year, publishing its first crawls in 2011. By 2015 the corpus held 1.8 billion webpages, and AWS support kept computing costs to an average of $2,000-4,000 per month at that time.[2]
Until 2023 it was funded almost exclusively by the Elbaz Family Foundation Trust, after which donations from the AI industry became significant. Rich Skrenta serves as executive director.[2]
Compliance And Controversy
The foundation's relationship with publishers has been contested. In November 2025, an investigation by The Atlantic reported that Common Crawl misled publishers about respecting paywalls and about honoring content-removal requests, showing no entries in its public search for sites that had opted out while still including them in scrapes used by AI companies; CCBot was found to be the most widely blocked bot among the top 1,000 websites.[2]
A 2024 study cited by Wikipedia found that 45% of C4's content was restricted by websites' terms of service for uses such as for-profit AI training, illustrating the legal ambiguity surrounding web-scale corpora.[2]
More datasets from Common Crawl
Common Crawl lists 1 datasets on this encyclopedia. The most representative listings:
More about Free & Open Data
Free & Open Data is one of the signal families covered by The Alternative Data Encyclopedia, with 36 providers and 322 datasets listed in the register. For how the signal is generated, its caveats and the compliance considerations, and the full provider list, see the category article.
Further reading
- How funds evaluate alternative data vendorsEvaluating alternative-data providers: research fit, provenance, historical timing, permitted use and the evidence needed for an informed purchasing decision.
- Open vs commercial alternative dataWhat free and open datasets can and cannot do: where public data rivals paid panels, where it cannot compete, and how open sources fit a professional research workflow.
- Alternative data for hedge fundsA practical guide to alternative data for hedge funds: team design, budget sequencing, infrastructure, vendor portfolio construction and the metrics that show whether the program creates investment value.
- Free alternative data sourcesThe practical list of free alternative-data sources for investors: public filings, official statistics, satellite and earth observation, on-chain data, web archives and open registries, what each covers, what it costs in engineering effort, and where free data falls short.