Which AI crawlers should a publisher allow?

Access policy starts with purpose

This guide helps a technical publisher set public-web crawler policy. Signal Studio’s objective is broad discoverability, so its public book pages, guides, resources, robots file, sitemap, and llms.txt are intended to be accessible to legitimate crawlers. Nonpublic publication assets and administrative paths remain protected by actual access controls.

Why one allow-or-block switch is too coarse

“AI crawler” combines several activities: conventional search indexing, answer retrieval, a user asking a product to open a page, model training, dataset collection, safety analysis, and unknown automation. Providers may publish several user agents because these purposes and controls differ.

Two simplistic policies follow. Blocking everything can undermine the publisher’s discovery goal. Allowing everything without path boundaries can expose licensed previews, unpublished manuscripts, query endpoints, or costly dynamic resources. Purpose and content class must be considered together.

Public guides open, manuscript assets closed

Search discovery, answer discovery, and user-triggered fetchers may access public HTML and images. Unpublished manuscripts, source archives, admin paths, and private preview assets remain unavailable regardless of crawler identity.

Path controls remain useful even under a broad public-content policy:

User-agent: *
Allow: /
Disallow: /private/
Disallow: /manuscripts/
Disallow: /admin/

The example expresses asset boundaries, not a universal policy. Named crawler directives should be added only after verifying current identifiers and deciding what each purpose may access.

Map agents, assets, rights, and operational cost

Inventory the site’s routes and classify them: public canonical content, public assets, user-specific content, administration, staging, licensed samples, unpublished files, search/query endpoints, and generated variants. Verify that private classes are actually authenticated.

Then inventory crawler directives from current provider documentation. Record user agent, stated purpose, verification method, desired policy, and last review. Compare edge logs with the declared list, but treat user-agent strings as claims rather than proof.

Publish explicit purpose-based controls

Start from the publishing objective. For maximum public discovery, allow documented legitimate crawlers on canonical public content and necessary rendering assets. Keep canonical HTML consistent for every crawler and human; do not serve deceptive crawler-only text.

Apply explicit disallows to paths that should not be fetched, while moving truly private resources behind authentication. Rate-limit abusive behavior at the edge based on verified signals and load, without assuming every unknown bot is a known provider.

Version robots policy and test it after deployment. Monitor response codes, crawl concentration, bandwidth, referrals, and indexing/citation observations. Revisit provider docs on a schedule and after a crawler announcement.

Crawler-purpose matrix

Maintain this matrix:

User agent / fetcher	Stated purpose	Public content	Private/licensed content	Verification	Monitoring	Last checked
Search discovery	Index and surface pages	Allow if desired	Authenticate/block	Provider method	Crawl/referral
Answer discovery	Retrieve for generated answers	Allow if desired	Authenticate/block	Provider method	Crawl/citation
User-requested fetch	Open a URL at user request	Allow public routes	Enforce user auth	Request/provider signals	Fetch/errors
Training crawler	Collect eligible public content	Allow or disallow by policy	Authenticate/block	Provider docs	Crawl volume
Unknown automation	Undeclared	Risk/load policy	Deny private paths	Network/behavior	Abuse/load

For each row, link the official documentation and record the exact robots group deployed.

Robots policies copied without an editorial decision

• Using one policy for public articles and private manuscript assets.
• Assuming a crawler name proves identity or purpose.
• Blocking rendering assets required to understand public pages.
• Treating robots.txt as authentication, copyright license, or enforceable data deletion.
• Changing crawler policy without checking production response codes and sitemap access.