Blog

Crawlers · Oct 26, 2026 · 5 min

AI2Bot, MistralAI-User, DeepSeekBot and QwantBot

Compare AI2Bot, MistralAI-User, DeepSeekBot and QwantBot through documented purposes, request headers, address checks, and Aiola Analytics categories.

Crawlers · Oct 26, 2026 · 5 min

Bytespider and TikTokSpider: ByteDance's Crawlers

Identify Bytespider and TikTokSpider traffic using published crawler details, address checks, robots.txt rules, and Aiola Analytics setup.

Crawlers · Oct 26, 2026 · 5 min

CCBot: Common Crawl's Crawler and Your robots.txt

Common Crawl's CCBot builds a public archive; its user-agent string, IP ranges, and robots.txt rules identify and manage visits.

Crawlers · Oct 26, 2026 · 5 min

ChatGLM-Spider, KimiBot and TimpiBot: AI Crawlers You May Not Know

Identify ChatGLM-Spider, KimiBot and Timpibot through their crawler records, published policies, user agents and robots.txt rules.

Crawlers · Oct 26, 2026 · 4 min

ClaudeBot, Claude-SearchBot and Claude-User Explained

Learn how ClaudeBot, Claude-SearchBot, and Claude-User differ, then match identity, set robots.txt, and read visits in Aiola Analytics.

Crawlers · Oct 26, 2026 · 5 min

Google-Extended: What It Controls and What It Does Not

Google-Extended sets Gemini training and grounding permissions, while Google Search indexing and AI Overviews use separate controls.

Crawlers · Oct 26, 2026 · 5 min

Google-GeminiNotebook: What It Fetches and How to Track It

Google-GeminiNotebook fetches Gemini Notebook sources, generally ignores robots.txt, and can show in Aiola Analytics under Answers.

Crawlers · Oct 26, 2026 · 6 min

GPTBot User Agent: Verify, Allow or Block It

Check OpenAI crawler identities and published IP ranges, choose training or search access, and review requests before changing your site rules.

Crawlers · Oct 26, 2026 · 6 min

How to Block Baiduspider (and When You Should Not)

Block Baiduspider with robots.txt or a WAF rule. Verify the caller through reverse DNS and understand what Baidu can still display.

Crawlers · Oct 26, 2026 · 5 min

ImagesiftBot: What It Crawls and How to Control It

See what ImagesiftBot collects from public pages, which robots.txt rules Hive documents, and which hosted crawler hits Aiola can store.

Crawlers · Oct 26, 2026 · 5 min

Meta-ExternalAgent, Meta-ExternalFetcher and Meta-WebIndexer

Understand Meta's three crawler roles, choose robots.txt rules for each, and see how their visits appear separately in Aiola Analytics.

Crawlers · Oct 26, 2026 · 5 min

OAI-SearchBot vs GPTBot vs ChatGPT-User

Compare OpenAI's search, training, and user-fetch bots, choose separate robots.txt rules, and review each bot in Aiola Analytics.

Crawlers · Oct 26, 2026 · 5 min

PetalBot: Huawei's Crawler, Verified and Controlled

Identify Huawei's search crawler, check its identity, choose crawling rules, and follow its requests alongside human traffic in Aiola Analytics.

Crawlers · Oct 26, 2026 · 5 min

QwenBot, Qwen-User, TongyiBot and AliyunBot

Compare Alibaba crawler records, robots.txt rules, and manual tracking steps for QwenBot, Qwen-User, TongyiBot, and AliyunBot in Aiola Analytics.

Crawlers · Oct 26, 2026 · 5 min

Verify Googlebot by IP: Google's Published Ranges

Check a request address against Google's published crawler ranges or use reverse and forward DNS before trusting its header.

Crawlers · Oct 26, 2026 · 5 min

What Is Meta-WebIndexer and Should You Allow It

Learn how Meta-WebIndexer supports Meta AI search, how related crawlers differ, and which robots.txt rules allow or block them.