Baiduspider collects pages and builds Baidu's search index. You can limit the whole site or selected paths in robots.txt, or reject a matching request at a WAF. Confirm the caller before you trust the User-Agent.
When a block changes Baidu search
Baidu's crawler help describes Baiduspider as the program that collects pages and builds the index. Its webmaster FAQ says those indexed pages are what search can return.
The FAQ lists a user-agent for each Baidu product. PC search and mobile search both use Baiduspider. Image search uses Baiduspider-image. News search uses Baiduspider-news. Video search uses Baiduspider-video. Baidu says that video token does not currently support robots.txt rules.
Baidu says this group blocks crawling from Baidu:
User-agent: Baiduspider
Disallow: /Baiduspider-cpro and Baiduspider-ads do not follow robots.txt. Baidu says those two are not added to the search index. A Baiduspider-image group can still allow a path, such as /image/, beside the broader block.
Baidu says a robots.txt change can take about 48 hours to stop new crawling. Its DNS data also refreshes on a cycle, so crawling can continue until then. Pages already in the index can take several months to disappear. For urgent removal, the FAQ points to Baidu's webmaster feedback form.
Baidu's robots.txt page describes another case. A blocked URL can still appear when another site links to it. Baidu will not crawl, index, or show that page's content. The visible text can be the other site's description.
Block Baiduspider in robots.txt
Serve robots.txt at the site root. Baidu checks that file before it crawls the host. Add the group above to the file you already serve. Leave your other user-agent groups in place.
A missing file, or an empty file, leaves the site open to crawlers. Create one only when you need to exclude something. Each group starts with User-agent, then Allow or Disallow lines. Put every excluded path on its own Disallow line. One line that lists two paths does not apply both.
Disallow is a prefix. Disallow: /help also matches /help.html, /helpabc.html, and /help/index.html. Disallow: /help/ matches /help/index.html and does not match /help.html. Check the paths your site really serves. Baidu reads Allow and Disallow from the top, and the first match wins. Baiduspider also accepts * and $ in a path. * matches any run of characters, and $ matches the end.
Fetch the live robots.txt on the host that received the request. Confirm the Baiduspider group matches the policy you published.
Baidu says the crawler has the same access as an ordinary visitor. It cannot fetch a page a visitor cannot open. Aiola's robots.txt guide treats Disallow as a request to a cooperative crawler. Keep private pages behind authentication.
Check the IP before you trust the name
Start with the source IP on your server or edge log. Keep the path, status, and full User-Agent with it. Baidu's webmaster FAQ says to reverse-resolve that IP. On Linux, run host. On Windows, run nslookup. On macOS, run dig -x. A real hostname ends in .baidu.com or .baidu.jp. Any other name is fake.
Baidu's host example returns baiduspider-123-125-66-120.crawl.baidu.com. Treat that name as an example, not a permanent allowlist. A name that only contains "baidu" fails the suffix test.
Aiola's Baiduspider reference adds a forward lookup. Baidu's pages do not require that second query. A PTR record alone can be controlled by an attacker.
- Reverse-resolve the recorded source IP.
- Require the hostname to end in
.baidu.comor.baidu.jp. - Resolve that full hostname forward.
- Require the forward result to include the original IP.
Aiola's page shows both steps with host. If either lookup fails, leave the request unconfirmed. Do not allow it only because the header says Baiduspider.
Aiola lists this sample User-Agent: Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html). A Developer Center article prints the same sample. That article is not Baidu's crawler help. Use it to find log lines, then decide identity with DNS.
Block the header at your WAF
Baidu says Baiduspider follows robots.txt. That file does not reject the connection. Use a WAF when the request itself must fail. Keep the robots.txt group as the stated preference, and add the edge rule separately.
Cloudflare recommends custom rules instead of User Agent Blocking rules. Its example matches http.user_agent with eq, and the action can be Block. String comparison is case-sensitive unless you change case. The lower() function converts a string field to lowercase.
A bot-rule page shows http.user_agent contains with Block. Bot Management on that page requires Enterprise. The expression below does not use a bot score. It joins lower(), contains, and http.user_agent. Cloudflare did not publish it as a Baiduspider rule. Review it in your zone before you deploy it.
lower(http.user_agent) contains "baiduspider"Set the action to Block. The rule matches header text and does not run the DNS checks. For a partial block, add starts_with on http.request.uri.path. Then compare the rule's matches with the paths you meant to stop.
Compare logs with search results
After you publish, read the live robots.txt again. If a WAF rule is active, review the requests it matched. Compare a similar window before and after the change.
A missing search listing is not enough proof. Check links from other sites, the 48-hour crawl stop, and the multi-month index wait separately. Logs show paths and statuses. A search check shows exposure.
If requests continue, test one caller before you widen the block. Repeat the suffix test and the forward lookup. A failed DNS check means the name is still unconfirmed.
Read Baiduspider rows in Analytics
Aiola Analytics includes Baiduspider in Indexing. The Analytics reference describes the AI view.
The AI view shows crawl totals, bots, pages, and activity over time. It groups hits into Answers, Indexing, Training, and Other. AI-answer referrals, crawler 404s, and discovery-file health also appear there.
Crawlers do not run JavaScript, so the browser tag cannot see them. One server or edge snippet from the setup guide posts each matching hit. Only known crawler user agents are stored.
A recorded crawl is not proof that the vendor cited the page. An empty period is not proof that the vendor cannot reach the site.
Try it in Aiola
Download Aiola free for macOS 12 or later and Windows 10 or later. Hosted Analytics needs Pro or the 7-day trial, and the pricing page lists both. Open the AI view and compare Baiduspider paths with your published rules.
