Common Crawl · Training
CCBot
Identify CCBot by its user agent. Check request identity, review robots.txt controls and track visits in Aiola Analytics.
Key facts
CCBotWhat CCBot is
CCBot builds Common Crawl's publicly accessible web-crawl repository. Common Crawl publishes IPv4 and IPv6 ranges and supports forward-confirmed reverse DNS under crawl.commoncrawl.org for IPv4.
In Aiola's taxonomy, Training means scraped for model training or dataset development.
CCBot user agent
Look for the match token inside the complete HTTP User-Agent header. Tokens can be spoofed, so use the network checks below for authentication.
CCBot/2.0 (https://commoncrawl.org/faq/)CCBotOfficial IP ranges
Common Crawl publishes an official IP range list. This cached snapshot contains 5 CIDR prefixes, last fetched 2026-10-23. Open the live source
18.97.14.80/29
18.97.14.88/30
18.97.9.168/29
2600:1f28:365:8000::/56
3.41.188.32/29Verify a CCBot request
For CCBot, compare the source IP from your trusted edge or server against Common Crawl’s current published prefixes.
python3 -c 'import ipaddress; print(ipaddress.ip_address("REQUEST_IP") in ipaddress.ip_network("18.97.14.80/29"))'Test every current prefix in the vendor feed and re-fetch it regularly. This cached snapshot is evidence of publication, not a permanent firewall list.
Control CCBot with robots.txt
Common Crawl documents crawler controls for this bot family. Robots.txt can direct cooperative crawling, but it cannot prove that a request is authentic.
User-agent: CCBot
Allow: /User-agent: CCBot
Disallow: /Aiola Analytics
See every CCBot visit on your site
Aiola tracks 70 AI, search and link-preview crawlers next to your human traffic. See when CCBot arrives, which pages it requests and how that changes over time. Hosted Analytics needs Pro or a trial.
Common questions
Build, ship and fix in one app
Errors, user feedback, GitHub and analytics live next to your coding agents in one desktop app. Every issue becomes a task an agent can take.