Dataset

Petabyte web crawl corpus

Monthly petabyte-scale crawls of the open web published as WARC segments with a CDX index service for targeted retrieval: raw material for language models, market mapping and large-scale entity extraction.

Visit provider

Dataset card

The Petabyte web crawl corpus is a publicly archived collection of raw web pages distributed by Common Crawl. It provides bulk raw HTML and metadata harvested from billions of URLs for large-scale text and NLP research.

What this dataset contains

The corpus consists of raw web pages and associated crawl metadata captured by Common Crawl's periodic web crawls. It is published in petabyte-scale archives of WARC and WAT files containing fetched HTML, HTTP headers, and URL-level metadata. The publisher distributes the data under CC0 / open terms for unrestricted redistribution and reuse.

Structure and fields

Files are organized as crawl segments indexed by crawl identifier and timestamp, with each record representing a single HTTP response and its fetched payload. Standard Common Crawl fields include URL, fetch time, HTTP response headers, content type, and raw HTML content; richer metadata such as extracted hyperlinks and language hints appears in companion WAT and WET index files. Detailed schemas are maintained in the publisher's documentation rather than in the dataset index.

How analysts use it

Quantitative researchers use the corpus to train large language models, build web-scale embeddings, and study long-run shifts in online content and language. Alternative-data teams derive signals from text such as brand or product mentions, hiring language on career pages, sentiment around listed companies, and technology adoption indicators inferred from site fingerprints. Because the corpus is raw and unstructured, downstream pipelines typically parse HTML, deduplicate pages, and extract the specific fields relevant to a given investment thesis.

Documentation & references 2