Automation & Web Scraping blog

Case study

What Common Crawl shipped with its June 2024 archive

A downloadable snapshot, an index and lists of files gave researchers several ways into a crawl containing billions of pages.

One crawl with a name and a date

Common Crawl published CC-MAIN-2024-26 on June 28, 2024. Its announcement covers captures made from June 12 through June 26 and reports 2.7 billion web pages. Of those URLs, 945 million had never been visited by Common Crawl before. For someone studying a change over time, the collection dates and the snapshot name identify which observations are in scope.

Choosing what to download

The release includes indexes and downloadable lists of archive files. Common Crawl also documents three ways to consume a capture. WARC files hold the archived records, WAT files expose calculated metadata, and WET files contain extracted plain text. Someone counting words and someone studying response metadata can work from the same release while fetching different files.

A lesson for smaller data projects

Give a recurring export a permanent name and a small manifest listing its input, capture window and output files. Imagine a customer asks why a product description differs from last month's report. With two named exports, you can locate the observations behind both answers. Keep the manifest next to the data so that a teammate receiving only the download can still identify the snapshot and choose the file they need.

Sources

All articles and case studies Describe your task