YandexBot

Yandex's primary search crawler. Aiola classifies it as indexing because it builds a searchable retrieval index.

In Aiola's taxonomy, Indexing means crawled for ai or search indexing.

Look for the match token inside the complete HTTP User-Agent header. Tokens can be spoofed, so use the network checks below for authentication.

Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

YandexBot

Yandex supports reverse-DNS verification for this crawler family. Accepted suffixes in this snapshot: .yandex.ru, .yandex.net, .yandex.com.

Use a double lookup: reverse-resolve the request IP and check the exact hostname suffix. Then forward-resolve that hostname and require the original IP. Checking only the visible hostname is vulnerable to attacker-controlled PTR records.

For YandexBot, reverse-resolve the source IP recorded by your trusted edge or server, then check the documented suffixes.

host REQUEST_IP
# Confirm the hostname ends in an official suffix
host RETURNED_HOSTNAME
# The forward lookup must return REQUEST_IP

Resolve the verified hostname forward and require the original IP in the result. A PTR hostname alone can be controlled by an attacker.

Yandex documents crawler controls for this bot family. Robots.txt can direct cooperative crawling, but it cannot prove that a request is authentic.

User-agent: YandexBot
Allow: /

User-agent: YandexBot
Disallow: /

Read official crawler documentation