How do you let Claude read and cite your website?

Most robots.txt files were written for one Claude crawler. Anthropic now documents three, each with a different job, and blocking the wrong one silently removes you from live Claude answers. Everything below is sourced from Anthropic's own crawler documentation and API documentation, read on 1 September 2026; where Anthropic has published nothing, we say so.

1. Which crawlers are Claude's, and what does each one do?

Anthropic's help center documents three user agents. ClaudeBot collects web content for model training: "ClaudeBot helps enhance the utility and safety of our generative AI models by collecting web content." Claude-User fetches pages during live conversations: "Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites." Claude-SearchBot "navigates the web to improve search result quality for users." Three agents, three consequences: training presence, live-answer presence, and search-result presence.

2. What should robots.txt allow?

A site that allows only ClaudeBot has opted into training data while staying invisible to the two agents that put you in front of a live buyer. If you want Claude to recommend you, the minimum is allowing Claude-User and Claude-SearchBot; each is controlled by a standard robots.txt user-agent block, and Anthropic honors the non-standard Crawl-delay directive if load worries you. Verification is possible: Anthropic publishes its crawler IP addresses at claude.com/crawling/bots.json, so a hit claiming to be Claude can be checked against the real list.

3. Does Claude have its own search index?

Anthropic does not say which index powers Claude's web search. The API documentation describes the capability ("direct access to real-time web content") without naming a provider, and the existence of a dedicated Claude-SearchBot implies retrieval infrastructure of Anthropic's own. Press reports have named third-party providers; none of that is in the official documentation, so we treat it as unconfirmed.

4. How does Claude cite sources?

The citation format is documented precisely. A web search result carries the page URL, its title, and the exact cited text span. Anthropic's API documentation also sets a display rule for builders: "When displaying API outputs directly to end users, citations must be included to the original source." The practical consequence for a manufacturer: when Claude uses your page, the citation carries your exact sentence. Pages written in short, complete, quotable statements give the engine something to carry; pages of adjectives do not.

5. What has Anthropic not told publishers?

There is no official Anthropic guidance on how to be cited: no page-structure recommendations, no metadata advice, no equivalent of Google's documentation for site owners. In that void we apply what is measurable everywhere else: machine-readable HTML, one consistent entity sentence, a page per capability, structured data, and per-engine measurement of what actually gets cited. That discipline is engine-agnostic by design, which is the point when one engine documents nothing.

What can this honestly promise?

Nobody can guarantee that Claude cites you; retrieval depends on what a buyer asks, and Anthropic publishes no selection criteria. What robots.txt hygiene guarantees is narrower and real: you are readable when the question comes. We keep our own sites open to all three agents and measure citations across engines under a written protocol, before and after.

Author: Sebastian Larsen, larsen.studio, Copenhagen. Sources: Anthropic crawler documentation (support.claude.com) and Claude API web-search and citations documentation (platform.claude.com), both read 1 September 2026. The free look checks exactly this class of problem: start with the free look.