Agent Analytics
Crawler Verification
Crawler verification process
A service name in a User-Agent does not establish that the request came from that service. Verification checks DNS, published IP ranges, ASN information, and behavioral patterns against the claimed crawler identity.
Why verification matters
- Distinguish impersonation
- Improve analytics accuracy
- Understand resource use
- Evaluate access against security requirements
- Improve delivery to legitimate crawlers
| Method | What it checks |
|---|---|
| Reverse DNS | Hostname and organization domain associated with the IP |
| IP ranges | Membership in published crawler address ranges |
| ASN | Source network ownership |
| Heuristics | Characteristic signatures and access patterns |
A shared-cloud ASN, User-Agent, or behavioral pattern alone cannot establish service authenticity. The list below describes verification methods; it does not imply equal verification confidence for every crawler.
Platform-specific verification
Platforms verifiable through published information
- OAI-SearchBot: IP range verification + ASN verification
Partially verified platforms
Some platforms use common cloud provider IPs or don't publish verification methods, making complete verification challenging:
- You.com
- Bytedance
- Yahoo
- OpenClaw
- DeepSeek
- Baidu
- BaiduSpider: Reverse DNS verification
- Huawei
- Yandex
- Gemini
Stay updated
Review verification methods as platforms emerge, crawler infrastructure changes, new methods become available, and security requirements evolve. Do not treat a fixed list as sufficient for future traffic.
Important note about data updates
Changes to identification methods can change historical and current classifications. When traffic shifts substantially, distinguish actual volume changes from verification or classification updates.
Crawler breakdown report
POST /v1/reports/crawler-breakdown is an Alpha read of stored visit logs.
Group by path, host, status, purpose, crawler, or UTC date.
Only requested dimensions are returned. Purpose is the stored bot type with the
existing crawler classification as fallback: ai_training, index, ai_agent
or unknown. A purpose describes a request, not proof of a citation or training.
Supply inclusive start_date and end_date. Optional filters accepts arrays
for path, host, status, purpose, and crawler. Values in a field use OR;
fields use AND. An empty array selects no visits. Unsupported fields and
dimensions return 400. Results are restricted to your workspace, ordered by
count and then the requested dimensions. limit, offset, has_more andnext_offset page the ranking. Historical daily counters have no path or
status evidence and are excluded.
orc reports crawler-breakdown --start-date 2026-11-01 --end-date 2026-11-02 --dimensions path,host,status,purposeMCP: get_crawler_breakdown_report accepts the same JSON fields.