---
title: Crawler Verification
description: Understand verification methods and identification limits for AI
  and search crawlers.
canonical_url: https://orchestor.io/docs/en/traffic/crawler-verification
markdown_url: https://orchestor.io/docs/en/traffic/crawler-verification.md
contentType: how-to
---

# Crawler Verification

## Crawler verification process

A service name in a User-Agent does not establish that the request came from that service. Verification checks DNS, published IP ranges, ASN information, and behavioral patterns against the claimed crawler identity.

## Why verification matters

- Distinguish impersonation
- Improve analytics accuracy
- Understand resource use
- Evaluate access against security requirements
- Improve delivery to legitimate crawlers

## Primary verification techniques

| Method | What it checks |
| --- | --- |
| Reverse DNS | Hostname and organization domain associated with the IP |
| IP ranges | Membership in published crawler address ranges |
| ASN | Source network ownership |
| Heuristics | Characteristic signatures and access patterns |

> **Warning**
> 
> A shared-cloud ASN, User-Agent, or behavioral pattern alone cannot establish service authenticity. The list below describes verification methods; it does not imply equal verification confidence for every crawler.

## Platform-specific verification

### Platforms verifiable through published information

- [**Google**](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
- [**Microsoft Bing**](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0)
- [**Apple**](https://support.apple.com/en-us/119829)
- [**OpenAI**](https://platform.openai.com/docs/bots)
- OAI-SearchBot: IP range verification + ASN verification
- [**Duck Duck Go**](https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot)
- [**Perplexity**](https://docs.perplexity.ai/guides/bots)
- [**Meta**](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/#identify-2)
- [**Amazon**](https://developer.amazon.com/amazonbot)

### Partially verified platforms

Some platforms use common cloud provider IPs or don't publish verification methods, making complete verification challenging:

- [**Anthropic**](https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- **You.com**
- **Bytedance**
- [**Common Crawl**](https://commoncrawl.org/ccbot)
- **Yahoo**
- **OpenClaw**
- **DeepSeek**
- **Baidu**
- BaiduSpider: Reverse DNS verification
- **Huawei**
- [**Mistral**](https://docs.mistral.ai/robots)
- **Yandex**
- **Gemini**

## Stay updated

Review verification methods as platforms emerge, crawler infrastructure changes, new methods become available, and security requirements evolve. Do not treat a fixed list as sufficient for future traffic.

## Important note about data updates

Changes to identification methods can change historical and current classifications. When traffic shifts substantially, distinguish actual volume changes from verification or classification updates.

## Crawler breakdown report

`POST /v1/reports/crawler-breakdown` is an Alpha read of stored visit logs.
Group by `path`, `host`, `status`, `purpose`, `crawler`, or UTC `date`.
Only requested dimensions are returned. Purpose is the stored bot type with the
existing crawler classification as fallback: `ai_training`, `index`, `ai_agent`
or `unknown`. A purpose describes a request, not proof of a citation or training.

Supply inclusive `start_date` and `end_date`. Optional `filters` accepts arrays
for `path`, `host`, `status`, `purpose`, and `crawler`. Values in a field use OR;
fields use AND. An empty array selects no visits. Unsupported fields and
dimensions return 400. Results are restricted to your workspace, ordered by
count and then the requested dimensions. `limit`, `offset`, `has_more` and
`next_offset` page the ranking. Historical daily counters have no path or
status evidence and are excluded.

```bash
orc reports crawler-breakdown --start-date 2026-11-01 --end-date 2026-11-02 --dimensions path,host,status,purpose
```

MCP: `get_crawler_breakdown_report` accepts the same JSON fields.

---

[Documentation index](https://orchestor.io/docs/llms.txt)