xAI · Training

xAI-Web-Crawler

Identify xAI-Web-Crawler by its user agent. Check request identity, review robots.txt controls and track visits in Aiola Analytics.

Download AiolaAI crawler directory

Key facts

User-agent token: xAI-Web-Crawler
Training: Scraped for model training or dataset development.
No IP verification is published for xAI-Web-Crawler; treat the token as a claim.

What xAI-Web-Crawler is

xAI-Web-Crawler is a broad xAI automated web-crawler identity. Aiola classifies it as training because it performs automated collection rather than a user-triggered fetch.

In Aiola's taxonomy, Training means scraped for model training or dataset development.

xAI-Web-Crawler user agent

Look for the match token inside the complete HTTP User-Agent header. Tokens can be spoofed, so use the network checks below for authentication.

Example User-Agent (unverified)
xAI-Web-Crawler
Case-insensitive match token
xAI-Web-Crawler

Official IP ranges

xAI has not published official IP ranges that Aiola successfully fetched for xAI-Web-Crawler. Aiola does not list cloud-provider ranges or community guesses as if they authenticated this crawler.

Verify a xAI-Web-Crawler request

No IP authentication method is documented in this record for xAI-Web-Crawler. Its user-agent token is a claim, not proof of identity.

Use the exact header token for classification and apply your own access controls. Check the vendor documentation for any newly published IP, token or header verification method.

Control xAI-Web-Crawler with robots.txt

xAI does not provide crawler documentation that Aiola could confidently verify for this token. The rules below express your preference, but vendor compliance is unconfirmed.

Allow xAI-Web-Crawler
User-agent: xAI-Web-Crawler
Allow: /
Block xAI-Web-Crawler
User-agent: xAI-Web-Crawler
Disallow: /

No official crawler documentation URL is listed because Aiola could not verify one confidently.

Aiola Analytics

See every xAI-Web-Crawler visit on your site

Aiola tracks 70 AI, search and link-preview crawlers next to your human traffic. See when xAI-Web-Crawler arrives, which pages it requests and how that changes over time. Hosted Analytics needs Pro or a trial.

Download AiolaSet up bot tracking
Aiola Analytics showing AI crawler visits by bot and page

Common questions

Build, ship and fix in one app

Errors, user feedback, GitHub and analytics live next to your coding agents in one desktop app. Every issue becomes a task an agent can take.

Download Aiola