Can AI assistants read your site? robots.txt and AI crawlers

When someone asks ChatGPT, Gemini or Perplexity which product to buy or which company to call, the answer is built from pages that the assistant's systems have read. If those systems cannot reach your site, they have very little to say about you, however good your pages are.

The first thing that decides whether they can is a small text file at the root of your site: robots.txt. Here is what it does, which AI crawlers exist, and how to decide what to allow.

AI crawlers are now a large part of bot traffic

In its review of 2025, Cloudflare reports that search engine crawlers made up 40% of verified bot traffic, and AI crawlers about half as much, 20%².

Not all AI crawling is the same. Over a twelve-month period measured by Cloudflare, 80% of AI crawling was for training, 18% for search and 2% for user actions¹. The group to watch is user actions: crawling for them rose more than 15x in 2025².

Three kinds of AI bots

It helps to sort AI crawlers by what they do with your pages:

• Training bots collect pages to train the models behind assistants.
• Search bots index pages so an assistant can find them and link to them. OpenAI's OAI-SearchBot, for example, is used to surface and link to websites in ChatGPT's search features².
• User action bots open a page at the moment someone asks a question. ChatGPT-User visits pages when people ask ChatGPT questions².

Some companies keep these jobs apart and others do not: Meta, OpenAI and Anthropic run separate bots for separate purposes, while Google and Microsoft each use a single crawler for several purposes¹. Where the bots are separate, you can say no to training and still say yes to search.

What robots.txt can and cannot do

The Robots Exclusion Protocol behind robots.txt is formally defined in RFC 9309, and it lets site owners tell crawlers which parts of a site they may access². The idea goes back to 1994, when it was first proposed as a way for site owners to control how crawlers access their content⁸.

It is a request, not a lock. The standard itself says its rules "are not a form of access authorization"⁸. Google puts it the same way: a robots.txt file cannot force a crawler to behave, it is up to the crawler to follow it⁷. Cloudflare calls it an honor system³.

Two practical points follow. robots.txt is mainly there to manage how much crawlers request from your site, and it is not the way to keep a page out of Google⁷. For private pages, use a password or a noindex rule instead.

Google: one crawler for Search, a separate token for Gemini

For Google Search, the control is the set of robots.txt rules for Googlebot, because AI is built into how Search works⁶. Block Googlebot and you leave Google Search altogether.

If you want a say over AI training and grounding in some of Google's other products, Google offers a separate token for that, Google-Extended⁶. It lets you decide whether content Google crawls from your site may be used to train future Gemini models⁵.

Most sites have not decided yet

Only about 37% of the top 10,000 domains seen by Cloudflare had a robots.txt file at all³. Of 3,816 top domains with such a file, 546, about 14%, had rules written for AI bots in particular⁴.

In 2025, AI crawlers were the user agents most often fully disallowed in robots.txt files, which means blocked from the whole site². Blocking is a valid choice. The risk is blocking without meaning to: a rule written against training bots, or a broad rule copied from a template, can also shut out the bots that put you in AI answers.

A short checklist

1. Open yoursite.com/robots.txt and read it. Make sure it does not block Googlebot or Bingbot by accident.
2. Decide on training and on search separately. If you do not want your content used for training, block the training bots by name and leave the search and user action bots allowed.
3. Do not rely on robots.txt to hide anything private. Use a password or noindex.
4. Check again after every site change. A new theme, plugin or firewall setting can change what crawlers see without anyone noticing.

The OLAB Assist free scan checks whether your robots.txt blocks the AI search crawlers and the assistants' user agents, and whether your firewall turns them away even when robots.txt allows them.