Meta
meta-webindexer
What it is
meta-webindexer indexes public pages for Meta's web-search and AI retrieval features. Aiola classifies it as indexing because it supports a searchable retrieval index.
In Aiola's taxonomy, Indexing means crawled for ai or search indexing.
User agent
Look for the match token inside the complete HTTP User-Agent header. Tokens can be spoofed, so use the network checks below for authentication.
meta-webindexermeta-webindexerOfficial IP ranges
Meta's network is identified here as AS32934. ASN ownership is a useful signal, not a bot-specific allowlist; combine it with the User-Agent and request behavior.
Verify authenticity
Start with the source IP recorded by your trusted edge or server, not an untrusted forwarded header.
host REQUEST_IP
# Confirm the hostname ends in an official suffix
host RETURNED_HOSTNAME
# The forward lookup must return REQUEST_IPFor production checks, test against every current prefix in the vendor feed. Re-fetch feeds regularly: a July 2026 snapshot is evidence of publication, not a permanent firewall list.
Control it with robots.txt
Meta documents crawler controls for this bot family. Robots.txt can direct cooperative crawling, but it cannot prove that a request is authentic.
User-agent: meta-webindexer
Allow: /User-agent: meta-webindexer
Disallow: /Track it
Aiola tracks this crawler on your site — crawls, pages, trends. See when meta-webindexer arrives, which URLs it requests, and how activity changes over time.
Explore Aiola AnalyticsFAQ
What is meta-webindexer?
meta-webindexer indexes public pages for Meta's web-search and AI retrieval features. Aiola classifies it as indexing because it supports a searchable retrieval index.
What user-agent token identifies meta-webindexer?
Match the case-insensitive token “meta-webindexer” in the User-Agent header. A matching header alone does not authenticate the sender.
Can I verify meta-webindexer by IP address?
Meta traffic can be checked against AS32934, but an ASN is broad. Combine network ownership with the exact user-agent token and your own risk controls.
Can robots.txt block meta-webindexer?
Meta documents crawler controls for this bot family. Robots.txt can direct cooperative crawling, but it cannot prove that a request is authentic. Use the specific “meta-webindexer” group when you want a rule for this identity without changing rules for every crawler.