𝗪𝗵𝗮𝘁 𝗶𝘀 𝗖𝗼𝗺𝗺𝗼𝗻 𝗖𝗿𝗮𝘄𝗹?
Common Crawl is an open repository of web crawl data with over 300 billion pages captured since 2008. It captures 6 to 6.5 billion web pages per quarterly crawl and freely provides downloadable archives to the public.
Common Crawl was one of the single largest training sources behind GPT-3 in 2020. Google used the C4 dataset to train Bard, which was later rebranded as Gemini. Major publishers such as the Associated Press, The New York Times, and Bloomberg have accused Common Crawl of scraping paywalled and subscriber-only news articles. The concern among these publishers is that AI companies may use this data to train LLMs.
This means brand positioning, claims, USPs, client stories, and thought leadership content can be baked into the model weights. The brand can then surface in answers through parametric memory without relying on RAG.
Common Crawl Datasets
The crawled corpus data is available in the form of WARC, WAT, and WET files. These file formats are used for feeding crawled data into language model training pipelines.
WARC (Web ARChive) files store the complete HTTP request, headers, and the original crawled HTML page.
WAT (Web Archive Transformation) files store structured metadata extracted from each WARC record, such as detected links, HTTP headers, content length, title, and other structural signals. The data is formatted as JSON metadata instead of the raw page.
WET (Web Extracted Text) files strip out HTML markup, scripts, and styling, leaving only the readable text content extracted from each page in the WARC file.
The Common Crawl corpus has been collected since 2008, and older datasets are available for download, with the crawl archive going back to the 2013 Crawl Archive of November 27, 2013.The latest crawl dataset was released on August 24, 2026, containing 2.14 billion web pages of uncompressed content.
5 Steps to Include Your Brand Content in AI Training Data via Common Crawl
Step1: Ensure CCBot Is Allowed to Crawl the Website
a)Audit the CDN and WAF settings for CCBot Blocks
The first gatekeeper for CCBot crawling your website is the CDN and WAF configuration. When a CCBot request hits the edge, a WAF rule, bot-management rule, or IP block can reject the request depending on how these controls are configured.
A blocked CCBot typically receives a 403 Forbidden response at the edge when it is blocked at the CDN level.
Popular CDNs such as Cloudflare can block bots through WAF rules and bot-management configurations.
I have seen many clients block bots by IP ranges or user-agent strings at the CDN level. The company I work with shares a list of bots sending requests to the domain every quarter and asks me which ones to allow or disallow. This is done to ensure that not all bots are allowed to scrape the website’s content.
This is why it is a best practice to reach out to your infrastructure or security team and access the CDN logs. Filter the logs by CCBot’s user-agent and known IP ranges, and look for 403, 429, or 503 responses.
- b) Explicitly add CCBot as a user agent and add an Allow directive in robots.txt file:
The robots.txt file is the second gatekeeper that can make or break an AI system’s ability to discover your content.
User-agent: CCBot
Allow: /
Alternatively, add an asterisk (*) wildcard that permits all crawlers and bots, including CCBot, to crawl the website.
User-agent: *
Allow: /
I have explicitly allowed CCBot on my website as well as on my clients’ websites. Like most bots, I hope CCBot chooses to honor its directives, so ensure that CCBot is not disallowed from crawling your website.
- c) Add XML sitemap links to the txt file to allow Common Crawl to discover and crawl the pages.
Step2: Get high harmonic centrality backlinks to Get AI to Surface Your Brand
Adding CC bot in robots.txt file doesn’t necessarily guarantee your pages in CC archives, let alone AI training data. Crawling is expensive, and not all organizations have the resources and time that Google and Bing have to crawl all publicly available websites, nor do they have infrastructure like Google Search Console. Therefore, common crawl has set a crawl budget limit.
To prioritize the crawling, CC employs a harmonic centrality algorithm. Yes,the key step is to gain visibility in sites that have high “Harmonic Centrality” scores. So what is harmonic centrality then ?
Harmonic Centrality measures how close a website is to all other websites in the web graph. A site that can be reached through fewer link hops from many other sites gets a higher HC score.
Harmonic Centrality scores are assigned based on how close your website is to the most connected nodes in the internet graph. For example, Wikipedia, Instagram, Facebook, YouTube, and Twitter are core nodes. The closer your website is to these nodes in the link graph, the higher its chances of being crawled by CCBot.
CC spends most of the crawl budget around websites that are central to the web graph. So in all practicality, if your website gets a backlink from the high HC score websites, the chances to get crawled is high.
*The cost associated with crawling and making datasets available for public download is significant. This might be one of the reasons Common Crawl randomly selects a portion of a website for crawling and storing in its dataset.
Start outreaching with top 1000 HC domains
Identify well-connected websites that are central to the link graph and promote your pages on these websites. To identify such websites, Metehan Yesilyurt has shared a list of the top 1,000 HC domains from a dataset of 121.1 million domains. He has also built a CC Checker tool where you can find referring links from any website.
Going by with the list, my enthusiasm went flat as not many sites out there need low effort. These are Press & media, Tech platforms, Authority orgs and ofcourse I found UGC & forums and Directories & listings and Social to salvage.
But with budget and long term strategy, we can leverage press and media, repurposing content on social platforms, seeking companies subject matter experts participating in UGC & forums. We can indeed get links from these sites but there is a tough road ahead.
Step3: Avoid Serving JavaScript-Dependent Content
CCBot currently doesn’t execute JavaScript. If your content is not available in the raw HTML, it may not become part of the Common Crawl archive. Similarly, your content may not be accessible to GPTBot, PerplexityBot, and ClaudeBot, potentially leading AI systems to hallucinate on branded prompts and reducing AI visibility for non-branded prompts.
Choose server-side rendering (SSR) or a pre-rendering service for JavaScript frameworks such as React, Vue, and Next.js to serve up-to-date content in the raw HTML.
Time to Get Into AI Training Data Is 3 to 9 Months
Common Crawl releases crawled data periodically. Getting links from websites with high HC scores may take one to three quarters on its own. Then comes the knowledge cutoff date of AI systems. The timeline below demonstrates why getting your content into AI training data requires a long-term plan:
- April 2025 — CCBot crawls your page
- June 2025 — Common Crawl makes the crawl data publicly available
- October 2025 — ChatGPT knowledge cutoff date
Your content missed the cutoff entirely. Now you are waiting for the next training cycle, its knowledge cutoff, and its release date. That could be another 6 to 12 months.
You have no control over any of these timelines. You can only control whether your pages are crawlable and whether you are consistently building links from high HC score sites.
