research web
orc research web discovers URLs, retrieves page bodies, follows links and extracts CSS-selected values or JSON-LD. Read saved inventories and capture history, or register and discover websites for a Workspace.
Authenticate the CLI and select an accessible Workspace. Use --workspace to set the scope for this call.
Usage
orc research web map --url https://example.com --fallback none --workspace <workspace-id> --jsonDiscover URLs with paid fallback disabled.
Subcommands
crawl
Follow links from a URL and retrieve pages synchronously, up to ten pages and depth three. This differs from the site screen’s batch acquisition of known URLs.
orc research web crawl [options]Unique options
--fallback
Body field: fallback; enum: none|firecrawl
Type: string. Optional.
orc research web crawl --fallback <value>--formats
Body field: formats; csv of: html|markdown
Type: string. Optional.
orc research web crawl --formats <value>--max-age
Body field: maxAge
Type: number. Optional.
orc research web crawl --max-age <value>--max-depth
Body field: maxDepth
Type: number. Optional.
orc research web crawl --max-depth <value>--url
(required) Body field: url; max 2048 chars
Type: string. Optional.
orc research web crawl --url <value>extract
Extract CSS-selected values or JSON-LD from one to ten URLs. Selectors mode requires one to twenty uniquely named selectors.
orc research web extract [options]Unique options
--fallback
Body field: fallback; enum: none|firecrawl
Type: string. Optional.
orc research web extract --fallback <value>--max-age
Body field: maxAge
Type: number. Optional.
orc research web extract --max-age <value>--mode
Body field: mode; enum: selectors|jsonld
Type: string. Optional.
orc research web extract --mode <value>--selectors
Body field: selectors; JSON array of objects (use --stdin for large resources)
Type: string. Optional.
orc research web extract --selectors <value>--urls
(required) Body field: urls; csv
Type: string. Optional.
orc research web extract --urls <value>history get
Read up to twenty saved capture versions for a URL within the thirty-day retention period.
orc research web history get [options]Unique options
--url
max 2048 chars
Type: string. Required.
orc research web history get --url <value>index get
Read a saved public URL inventory and its acquisition time. The result is null when no index is saved.
orc research web index get [options]Unique options
--url
max 2048 chars
Type: string. Required.
orc research web index get --url <value>map
Discover up to 100,000 URLs from sitemaps, llms.txt and links without retrieving all page bodies. Use --fallback none to disable the default Firecrawl fallback.
orc research web map [options]Unique options
--fallback
Body field: fallback; enum: none|firecrawl
Type: string. Optional.
orc research web map --fallback <value>--max-age
Body field: maxAge
Type: number. Optional.
orc research web map --max-age <value>--sitemap
Body field: sitemap; enum: include|only|skip
Type: string. Optional.
orc research web map --sitemap <value>--url
(required) Body field: url; max 2048 chars
Type: string. Optional.
orc research web map --url <value>scrape
Retrieve page bodies for one to ten URLs. Choose HTML or Markdown with --formats, or request fresh acquisition with --max-age 0.
orc research web scrape [options]Unique options
--fallback
Body field: fallback; enum: none|firecrawl
Type: string. Optional.
orc research web scrape --fallback <value>--formats
Body field: formats; csv of: html|markdown
Type: string. Optional.
orc research web scrape --formats <value>--max-age
Body field: maxAge
Type: number. Optional.
orc research web scrape --max-age <value>--urls
(required) Body field: urls; csv
Type: string. Optional.
orc research web scrape --urls <value>sites list
List websites registered in the selected Workspace.
orc research web sites list [options]sites create
Register a domain with an empty website index without starting acquisition or a schedule.
orc research web sites create [options]Unique options
--domain
(required) Body field: domain; max 253 chars
Type: string. Optional.
orc research web sites create --domain <value>sites discover
Find up to thirty unregistered site candidates linked from a homepage without registering them, saving captures or invoking paid fallback.
orc research web sites discover [options]Unique options
--domain
(required) Body field: domain; max 253 chars
Type: string. Optional.
orc research web sites discover --domain <value>Examples
Supply a structured CSS-selector request.
orc research web extract --urls https://example.com --mode selectors --selectors '[{"name":"title","selector":"h1","multiple":false}]' --workspace <workspace-id> --jsonSupply a structured CSS-selector request.
Inputs and acquisition bounds
--urls and --formats accept comma-separated values. --max-age uses seconds and --max-depth controls link depth. URLs cannot contain credentials or arbitrary ports.
orc research web scrape --urls https://example.com,https://example.com/docs --formats markdown --max-age 0 --workspace YOUR_WORKSPACE_ID
orc research web crawl --url https://example.com --limit 5 --max-depth 1 --fallback none --workspace YOUR_WORKSPACE_ID
orc research web extract --urls https://example.com --mode jsonld --workspace YOUR_WORKSPACE_ID
orc research web extract --urls https://example.com --mode selectors --selectors '[{"name":"title","selector":"h1","multiple":false}]' --workspace YOUR_WORKSPACE_ID
orc research web index get --url https://example.com --workspace YOUR_WORKSPACE_ID
orc research web history get --url https://example.com --workspace YOUR_WORKSPACE_ID
orc research web sites create --domain example.com --workspace YOUR_WORKSPACE_ID
orc research web sites discover --domain example.com --workspace YOUR_WORKSPACE_ID
orc research web sites list --workspace YOUR_WORKSPACE_ID- Map discovers up to 100,000 URLs from sitemaps, llms.txt and links. It does not retrieve every page body.
- Scrape and extract accept up to 10 URLs per call. Crawl traverses up to 10 pages and depth 3. CLI crawl is link traversal; the site screen's Crawl batches known URLs up to 1,000 pages.
- Extract parses CSS selectors or JSON-LD without LLM inference. Selectors mode requires 1–20 uniquely named selectors.
attributereads an attribute;multiple: truereturns multiple matches. Use--stdinfor complex JSON inputs. - To persist scraped bodies while returning metadata only, pass
"formats": []through JSON input. - Sites create only registers a domain; it starts no acquisition or schedule. Discover returns up to 30 unregistered sites linked from a homepage without registering them, saving captures or invoking paid fallback.
Responses, persistence and errors
JSON retains the full API response inside data. For acquisition, inspect each URL's status in data.data, plus data.truncated, data.warnings and data.usage. HTTP 200 may include partial failures or truncation. Usage counts requests, not currency. Sites and history return data.data; index returns data.result and data.fetchedAt, null when no saved index exists.
Acquisition persists content. Shareable anonymous captures may be reused for up to 24 hours. Query-bearing URLs, non-shareable responses and Firecrawl captures remain workspace-scoped. History returns up to 20 versions readable for 30 days. --max-age 0 requests fresh acquisition. Caller authentication cookies are never sent to source sites.
Map runs synchronously for up to 90 seconds; other acquisition operations are bounded to 40 seconds. Robots denials and private network restrictions are retained. Map's default Firecrawl fallback can invoke the external provider when native sitemap discovery is incomplete. Other primitives only allow provider fallback on blocked responses when explicitly selected.
--dry-run previews a request without sending or saving it. --raw removes the CLI envelope. Automatic traversal with --page-all is unavailable. Exit codes: success 0, API/authentication/network failure 1, argument error 2. For 401/403, check user login, Workspace and access requirements. Inspect per-URL outcomes before retrying partial failures.
Permissions
Authentication, operation scopes and access to the selected Workspace apply.
Global Options
The following global options can be used with orc research web:
For details and examples, see global options.