Investigating bot access
Fetch access data for a connected domain and compare it with citation data to choose pages to investigate.
When to use this workflow
“Are AI bots reaching important pages, and are referrals increasing?”
Before fetching
Fix the instrumented domain, important-page inventory, and dates. Treat training crawlers, search and answer bots, and human referrals as separate observations.
Quick reference
Use this sequence once you know the task. Read the numbered steps below for the reasoning and checks.
orc reports bots get --domain YOUR_REGISTERED_DOMAIN --start-date 2026-09-01 --end-date 2026-09-07 --json
orc sources urls list --jsonPrerequisites
Complete the CLI quickstart. These commands use CLI 0.5.0. Select the target workspace and use IDs returned by list commands.
A registered domain with bot measurement connected.
Use a domain with bot measurement already connected. Replace YOUR_REGISTERED_DOMAIN and the dates with your investigation scope. For this bot report, a date-only end date includes that day.
1. Read collected bot access
Replace the registered domain and dates with your scope. This date-only range includes the end day. Distinguish missing integration from no access.
orc reports bots get --domain YOUR_REGISTERED_DOMAIN --start-date 2026-09-01 --end-date 2026-09-07 --json2. Narrow the target URLs
Select URLs for updates or investigation and check the citation period. Fetch matching conditions if the comparison period differs.
orc sources urls list --json3. Verify and save the result
Match returned periods and URLs, then identify pages whose access or errors need investigation. Bot visits do not guarantee citations or human referrals. Distinguish an unconnected domain or incomplete measurement period from an absence of visits.
Separate bot roles from human referrals
Read hourly bot types and paths, and inspect human referrals in a separate report. Compare observed paths with the important-page inventory.
orc reports bots get --domain YOUR_REGISTERED_DOMAIN --granularity hour --dimensions path,bot_type --metrics count --start-date 2026-09-01 --end-date 2026-09-07 --json
orc reports referrals get --domain YOUR_REGISTERED_DOMAIN --dimensions path,referral_source --metrics visits --start-date 2026-09-01 --end-date 2026-09-07 --jsonConfirm log coverage before calling an absent path never crawled. Referral visits are a measured floor, not every AI-assisted visit or conversion. Diagnosing failures by HTTP status requires evidence that actually carries response status.
When data is missing
If data is missing, check the workspace, collection period, filters, and pagination. Keep a failed request separate from an empty result. Use --help to inspect supported options.
If a failure remains, report it and verify the fix, including this workflow and the failed step.