About Agent Analytics
About Agent Analytics
Agent Analytics examines which pages AI crawlers retrieve, when they retrieve them, and how they interact with the site. Server and CDN request logs reveal activity that JavaScript-based measurement can miss.
Why traditional analytics fall short
| Traditional measurement assumes | AI crawlers may behave differently |
|---|---|
| JavaScript execution | Retrieve content without executing JavaScript |
| Cookies and sessions | Do not maintain session state |
| Browser navigation | Use specialized crawlers |
| Predictable browsing patterns | Vary by indexing, training, or real-time assistance |
Architecture built for AI
- Server-side collection: collect requests handled by the infrastructure.
- CDN integration: observe requests served from CDN caches.
- Crawler identification: combine User-Agent information with IP, DNS, and other verification methods.
- Analysis: compare collected logs by time, page, and crawler.
Coverage depends on the source and logging configuration. Origin-only logs can miss requests served entirely from CDN caches.
AI platform coverage
AI platforms include OpenAI, Anthropic, Google, Perplexity, Microsoft, Apple, Meta, DeepSeek, and other systems. Identification methods and confidence vary by crawler; see Crawler Verification.
Implementation approaches
CDN integration
For sites using a CDN, connect through the provider’s log delivery feature. This captures requests served at the edge and uses the provider’s log format.
Direct server integration
Without a CDN, review the custom integration for sending server logs. For infrastructure-specific requirements, first identify the collection point and required fields.
Key features
AI crawler analytics
- Crawler identification and classification
- Crawl events over time
- Access patterns and trends
- Page coverage and gaps
Performance metrics
- Response time
- Content accessibility
- Cache effectiveness
- Geographic distribution of AI access