AI Crawlers, robots.txt and llms.txt: Which Bots Should You Allow?

What are AI crawlers, and how do they differ from traditional search bots?

AI crawlers are automated clients that fetch web pages to train a language model, to feed the index of an AI-powered search product, or to read, in real time, a page a user asks for in the middle of a conversation. A traditional search bot crawls a page so it can show it in a list of results; an AI crawler may use the content as raw material for an answer or for a model.

Technically, there is little difference between them: both request the page over HTTP, identify themselves with a user-agent name and, if they follow the rules, read the robots.txt file first. The real difference is where the content ends up. The same company often runs separate bots for different purposes, and in robots.txt each one has to be addressed by its own name (its user-agent token). That is why a single “block AI” rule so often has unintended consequences.

The topic now matters directly for sites in Türkiye too: Google announced that it began rolling out AI Mode and AI Overviews in Türkiye on February 18, 2026. According to Google, the prerequisite for being shown as a supporting link in these features is that the page is indexed in Search and eligible to be shown with a snippet. We explain how these features work in our guide to AI Overviews and AI Mode, and the content side of becoming a source in AI answers in What is GEO?

What is the difference between training, search and user-request bots?

Training bots collect content as training data for future models; search bots feed the index of AI-powered search products and make it possible for answers to show source links; user-request bots fetch a page on the spot when a user asks for it during a conversation. For visibility, the search and user-request bots matter most.

The table below is based on documentation published by the providers as of September 2026. Bot names and roles change from time to time, and new bots get added, so check the relevant documentation again before writing any rules.

Two details in the table cause the most confusion in practice. First, some user-request bots are not bound by robots.txt: OpenAI states that robots.txt rules may not apply to ChatGPT-User, and Perplexity says that Perplexity-User generally ignores them. The same is true of Google’s user-triggered fetchers. The way to stop them is a server or CDN rule, not robots.txt; but because these bots fetch a page the user has explicitly asked for, blocking them means your page cannot be read in a conversation where someone asks about your brand.

Second, the archive CCBot builds is published openly. Who later uses content that enters this archive, and for what purpose, is outside the site’s control; for a site that wants to stay out of training use, the CCBot decision matters at least as much as the GPTBot decision.

Which bots should you allow in robots.txt, and which can you block?

A site that wants to appear as a source in AI answers should allow the search bots; blocking the training bots is a content-policy decision that does not directly cut off visibility. According to OpenAI’s documentation, the two settings work independently: a site that blocks GPTBot but allows OAI-SearchBot can continue to appear in ChatGPT search results. Which bots to allow is a decision taken together with your content policy and business goals; in our SEO consultancy for AI search we settle it in line with the site’s objectives.

The thing to remember is that robots.txt is a request, not a lock. RFC 9309 (September 2022), which defines the standard, states plainly that these rules are not a form of access authorisation and that crawlers are asked to honour them. Bots that ignore the rules or disguise their identity have to be blocked at server, firewall or CDN level. The same standard recommends that crawlers should not use a cached copy of the file for more than 24 hours, except when the file cannot be reached, which is why the effect of a change may not show up straight away.

1. Write down the goal. Do you want to be a source in answers, to keep your content out of model training, or both? This is a decision for the content and business side, not the technical team.
2. Keep the search bots open. When Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot are blocked, visibility in the corresponding search products shrinks too.
3. Decide on training use separately. GPTBot, ClaudeBot and CCBot, along with the control tokens Google-Extended and Applebot-Extended, belong to this group. Blocking them does not switch off search visibility; it signals that you do not want your content used to train these models.
4. Write rules by bot name. Putting Disallow: / under a User-agent: GPTBot line blocks only that bot. Bots without a group of their own fall back to the User-agent: * group, and a broad Disallow there can catch the search bots too.
5. Don’t rely on robots.txt for private areas. Areas such as member dashboards, carts or customer documents should be protected by authentication; any path listed in robots.txt can be read by anyone.

On WordPress sites this file is often generated by a plugin or by WordPress itself; where to edit it is covered in our WordPress SEO guide. The general effect of robots.txt on crawling and indexing is explained in What is technical SEO?, and common problems such as rules that accidentally block the whole site are covered in our article on SEO mistakes.

What does Google-Extended control, and what doesn’t it?

Google-Extended is not a separate crawler but a control token used in robots.txt: it determines whether content crawled by Google’s existing bots may be used to train future Gemini models and for grounding (basing an answer on sources) in Gemini apps and Vertex AI. According to Google, it does not affect a site’s inclusion in Google Search and is not used as a ranking signal in Search.

So blocking Google-Extended does not stop a page from appearing in AI Overviews or AI Mode, because these features are part of Search. To limit the information shown from a page in these features, Google points to Search’s preview controls.

• Managed by Google-Extended: training of Gemini models; grounding with Google Search in Gemini apps and Vertex AI.
• Not managed by Google-Extended: inclusion in Google Search, rankings, and appearance in AI Overviews and AI Mode.
• Controls for AI Overviews and AI Mode: nosnippet, data-nosnippet, max-snippet and noindex. These also affect the classic search snippet: restrict the AI summary and you restrict the description in the regular result as well.

Apple and Microsoft draw a similar line. According to Apple’s documentation, Applebot-Extended does not crawl any pages; it only determines whether data collected by Applebot may be used to train Apple’s foundation models, and when it is blocked, content remains discoverable in Spotlight, Siri and Safari. Microsoft, for its part, announced in September 2023 that content carrying the noarchive meta tag would not be included in Bing Chat (today’s Copilot) answers or used to train its models, and that only the URL, title and snippet would be used from content carrying nocache; neither tag prevents the page from appearing in Bing search results.

What is an llms.txt file, and do you need one?

llms.txt is a proposed file, usually placed in a site’s root directory, that summarises the site’s most important pages in Markdown for language models and AI agents; it is not mandatory and, unlike robots.txt, contains no access rules. Google states clearly that no new machine-readable files or AI-specific text files are needed to appear in AI Overviews and AI Mode.

The proposal was published by Jeremy Howard on llmstxt.org in September 2024; a second version, revised in light of two years of real-world use, followed in August 2026. Under the specification, the only required part of the file is an H1 heading with the name of the site or project; it is followed by a short summary in a blockquote, optional notes and lists of links under H2 headings. The proposal also recommends serving clean Markdown versions of pages at the same address with .md appended, or with the extension replaced by .md.

• Where it can make sense: clearly structured, frequently consulted resources such as software documentation, API references and technical content that developers read through coding assistants.
• What not to expect: a ranking boost on Google, or automatically being cited as a source in AI answers. Those outcomes are determined by access, indexing and the content itself.
• The risk to watch: if the file is not kept up to date, it will contradict the information on the site. Rather than copying volatile details such as prices, service scope or contact information into it, it is safer to link to the relevant page.

In our own practice we treat llms.txt as a low-cost, optional extra; we do not use it as a substitute for robots.txt, an XML sitemap or indexability. If you add one, it should contain nothing other than information actually published on the site. Our llms.txt generator builds the draft from the real pages in your sitemap and shows the status of AI bots in your robots.txt.

How do you verify bot access in your server logs?

Whether a bot really belongs to the company it claims is verified from its IP address, not its user-agent name: run a reverse DNS lookup on the IP followed by a forward DNS lookup, or compare the IP against the list of IP ranges published by the company that operates the bot. A user-agent string is easy to spoof; Common Crawl itself notes that some crawlers falsely identify themselves as CCBot.

1. Segment your logs by bot name. Filter your server access logs for names such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot; work out which URLs they request, how often and with which status codes.
2. Verify with reverse and forward DNS. For Google, the reverse DNS result should end in googlebot.com, google.com or googleusercontent.com, and the forward lookup should return the same IP; for Bingbot the domain should be search.msn.com, and for Applebot applebot.apple.com.
3. Compare against the published IP lists. OpenAI publishes a separate JSON file for each of its bots (gptbot.json, searchbot.json, chatgpt-user.json); Perplexity, Anthropic, Common Crawl, Apple and Google also share their IP ranges as JSON.
4. Read the status codes. If a verified search bot is getting a 403, a 429 or a challenge page, the block is not in robots.txt; it sits in the server, a security plugin or the CDN layer.
5. Check again after any change. After changing a rule, review the logs again with the same filter; because robots.txt is cached, the effect may not be visible straight away.

CDN and firewall settings are the most frequently overlooked layer in all this. Cloudflare, for example, has since July 2025 asked new domains during setup whether AI crawlers should be allowed, with blocking as the default; in July 2026 it split these controls into three classes (search, agent and training) and left the search class enabled by default on new domains. Even so, an overly broad bot block or a hand-written firewall rule can stop search bots too, without leaving any trace in robots.txt. For crawl data on the Bing side and IndexNow, see our Google Search Console guide.

A free SEO and GEO analysis is a good starting point for seeing which bots your site is open to at the robots.txt, CDN and server layers; llms.txt and AI bot access are also among the GEO items in our SEO packages.