Which AI crawlers should a publisher allow?

Scope

This guide helps a technical publisher set public-web crawler policy. Signal Studio’s objective is broad discoverability, so its public book pages, guides, resources, robots file, sitemap, and llms.txt are intended to be accessible to legitimate crawlers. Nonpublic publication assets and administrative paths remain protected by actual access controls.

Why it happens

“AI crawler” combines several activities: conventional search indexing, answer retrieval, a user asking a product to open a page, model training, dataset collection, safety analysis, and unknown automation. Providers may publish several user agents because these purposes and controls differ.

Two simplistic policies follow. Blocking everything can undermine the publisher’s discovery goal. Allowing everything without path boundaries can expose licensed previews, unpublished manuscripts, query endpoints, or costly dynamic resources. Purpose and content class must be considered together.

Diagnosis

Inventory the site’s routes and classify them: public canonical content, public assets, user-specific content, administration, staging, licensed samples, unpublished files, search/query endpoints, and generated variants. Verify that private classes are actually authenticated.

Then inventory crawler directives from current provider documentation. Record user agent, stated purpose, verification method, desired policy, and last review. Compare edge logs with the declared list, but treat user-agent strings as claims rather than proof.

Solution

Start from the publishing objective. For maximum public discovery, allow documented legitimate crawlers on canonical public content and necessary rendering assets. Keep canonical HTML consistent for every crawler and human; do not serve deceptive crawler-only text.

Apply explicit disallows to paths that should not be fetched, while moving truly private resources behind authentication. Rate-limit abusive behavior at the edge based on verified signals and load, without assuming every unknown bot is a known provider.

Version robots policy and test it after deployment. Monitor response codes, crawl concentration, bandwidth, referrals, and indexing/citation observations. Revisit provider docs on a schedule and after a crawler announcement.

Artifact

Maintain this matrix:

User agent / fetcher	Stated purpose	Public content	Private/licensed content	Verification	Monitoring	Last checked
Search discovery	Index and surface pages	Allow if desired	Authenticate/block	Provider method	Crawl/referral
Answer discovery	Retrieve for generated answers	Allow if desired	Authenticate/block	Provider method	Crawl/citation
User-requested fetch	Open a URL at user request	Allow public routes	Enforce user auth	Request/provider signals	Fetch/errors
Training crawler	Collect eligible public content	Allow or disallow by policy	Authenticate/block	Provider docs	Crawl volume
Unknown automation	Undeclared	Risk/load policy	Deny private paths	Network/behavior	Abuse/load

For each row, link the official documentation and record the exact robots group deployed.

Common mistakes

• Using one policy for public articles and private manuscript assets.
• Assuming a crawler name proves identity or purpose.
• Blocking rendering assets required to understand public pages.
• Treating robots.txt as authentication, copyright license, or enforceable data deletion.
• Changing crawler policy without checking production response codes and sitemap access.