Common Crawl
Open Source Web Crawling Dataset
Common Crawl is a large-scale open-source web crawling dataset that provides raw data, metadata, and text extracts from billions of web pages every month, widely used for NLP, search engines, and large language model pre-training research. It includes 3 subsets: Full, Simplified Chinese, Traditional Chinese.
Dataset Highlights
Open data on an internet scale, driving the next generation of artificial intelligence research
Ultra-large Scale
Continuously crawled the internet since 2008, collecting billions of web pages each month, with a cumulative data volume reaching PB level, making it one of the largest publicly available web datasets in the world.
Multi-format Support
Provides three standard formats: WARC (original web pages), WAT (structured metadata), and WET (plain text extraction), to meet different research and engineering needs.
Chinese Subset
Specifically divided into simplified Chinese and traditional Chinese data partitions, providing high-quality native language corpus for Chinese NLP research and large language model training.
Free and Open
No registration or API key application required, data hosted on AWS S3, anyone can directly download and use it, truly public domain data.
NLP Cornerstone
The pre-training corpus for many large language models such as GPT, BERT, and T5 comes from Common Crawl, making it an indispensable data foundation for modern NLP research.
Regular Updates
New crawled data is released monthly, with consistent formatting, making it convenient for researchers to continuously access the latest internet content for temporal analysis and model iteration.
Applicable Scenarios
From model training to academic research, covering various data-driven applications
Pre-training of Large Language Models
As a core training data source for large language models like GPT, LLaMA, and BLOOM, providing a vast amount of multilingual text corpus
Web-level NLP
Supports large-scale natural language processing tasks such as language detection, named entity recognition, and sentiment analysis
Search Engine Research
Used to build and test web search algorithms, webpage ranking models, and information retrieval systems
Chinese Language Processing
Utilizing subsets of Simplified and Traditional Chinese for research in Chinese word segmentation, text classification, and semantic understanding
Data Preview
The following is a header example of WARC format data, showing the structure of a single web page crawl record
WARC/1.0 WARC-Type: response WARC-Date: 2024-03-15T08:23:45Z WARC-Target-URI: https://example.com/article Content-Type: application/http; msgtype=response Content-Length: 28456 WARC-Record-ID: <urn:uuid:a1b2c3d4-e5f6-7890-abcd-ef1234567890> HTTP/1.1 200 OK Content-Type: text/html; charset=UTF-8 <!DOCTYPE html> <html lang="zh-CN"> <head><title>Example Web Page</title></head> ...
3 Steps to Get Started Quickly
From browsing to usage, you can start your data research project in just a few minutes
Browse Datasets
View dataset details on the Ace Data Cloud platform and select the desired subset: Full, Simplified Chinese, or Traditional Chinese.
Download or Stream
Download or stream WARC, WAT, WET format files, with data hosted on high-speed storage, supporting resume downloads.
Parse and Use
Use tools like warcio to parse the data and extract text content for NLP pipelines or model training.
Start Exploring Common Crawl Data
An open dataset at internet scale, free to use, and regularly updated. Whether you are training large language models or conducting web data analysis, Common Crawl is an ideal starting point for your data.