Diffbot · Indexing
Diffbot
Identify Diffbot by its user agent. Check request identity, review robots.txt controls and track visits in Aiola Analytics.
Key facts
DiffbotWhat Diffbot is
Diffbot crawls pages for its search index and Knowledge Graph and excludes AI training. It follows robots.txt by default with agreed exceptions. Diffbot also documents Diffbot-User for requests made on behalf of a human user in response to their input.
In Aiola's taxonomy, Indexing means crawled for ai or search indexing.
Diffbot user agent
Look for the match token inside the complete HTTP User-Agent header. Tokens can be spoofed, so use the network checks below for authentication.
DiffbotDiffbotOfficial IP ranges
Diffbot has not published official IP ranges that Aiola successfully fetched for Diffbot. Aiola does not list cloud-provider ranges or community guesses as if they authenticated this crawler.
Verify a Diffbot request
No IP authentication method is documented in this record for Diffbot. Its user-agent token is a claim, not proof of identity.
Use the exact header token for classification and apply your own access controls. Check the vendor documentation for any newly published IP, token or header verification method.
Control Diffbot with robots.txt
Diffbot documents crawler controls for this bot family. Robots.txt can direct cooperative crawling, but it cannot prove that a request is authentic.
User-agent: Diffbot
Allow: /User-agent: Diffbot
Disallow: /Aiola Analytics
See every Diffbot visit on your site
Aiola tracks 70 AI, search and link-preview crawlers next to your human traffic. See when Diffbot arrives, which pages it requests and how that changes over time. Hosted Analytics needs Pro or a trial.
Common questions
Build, ship and fix in one app
Errors, user feedback, GitHub and analytics live next to your coding agents in one desktop app. Every issue becomes a task an agent can take.