The short answer
AI assistants find products through search crawlers that respect robots.txt, through pages users ask them to open, and through ordinary search indexes. Let the search crawlers in, put your key facts in HTML that loads without JavaScript, and make pricing and docs easy to fetch. In our check of 182 AI product sites, only 4 blocked an AI search crawler, but 13% served homepages that are nearly empty without JavaScript and another 13% turned away a plain automated request.
| Crawler | Company | Used for | Follows robots.txt | Sites blocking it (of 168) |
|---|---|---|---|---|
| GPTBot | OpenAI | Training data | Yes | 9 |
| OAI-SearchBot | OpenAI | ChatGPT search results | Yes | 3 |
| ChatGPT-User | OpenAI | Pages a user asks ChatGPT to open | May not apply | Not checked |
| ClaudeBot | Anthropic | Training data | Yes | 7 |
| Claude-SearchBot | Anthropic | Search quality in Claude | Yes | 2 |
| Claude-User | Anthropic | Pages fetched in response to a user question | Yes | Not checked |
| PerplexityBot | Perplexity | Perplexity search results | Yes | 3 |
| Perplexity-User | Perplexity | Pages fetched to answer a question | Generally not | Not checked |
| Google-Extended | Control token for Gemini training and grounding, not a separate crawler | Yes | 6 |
How AI assistants find products
AI companies use different bots for different jobs. OpenAI separates GPTBot training crawls from OAI-SearchBot search crawls; Anthropic separates ClaudeBot from Claude-SearchBot. PerplexityBot serves search and is not used to crawl content for AI foundation models. Training and search controls are separate: OpenAI says sites that block OAI-SearchBot are left out of ChatGPT search answers, though they may still appear as navigational links.
A third kind of request comes from users. OpenAI says robots.txt may not apply to ChatGPT-User requests. Perplexity says Perplexity-User generally ignores robots.txt. Anthropic says all its bots follow robots.txt, and disabling Claude-User prevents Claude from retrieving your content in response to a user query.
OpenAI says search access changes can take about 24 hours after a robots.txt update; Perplexity says its changes may take up to 24 hours. If GPTBot and OAI-SearchBot are both allowed, OpenAI may reuse one crawl for both purposes. Its OAI-AdsBot only visits pages submitted as ads.
Google's AI Overviews and AI Mode use the regular Search index. Google says there are no extra requirements and no special files or markup needed: a page that is indexed and eligible for a snippet can be used as a supporting link.
Google-Extended controls Gemini training and grounding through robots.txt and has no separate HTTP user agent. It does not affect inclusion or ranking in Google Search.
What we checked
On October 2, 2026 we checked the websites of 182 AI products in the SOTA catalog, counting each site once and leaving out products that live on a section of another company's site. For each site we read robots.txt and tested whether common AI crawlers may fetch the homepage, looked for /llms.txt and a sitemap, and fetched the homepage to record its title, description, share image, structured data and how much text it contains before JavaScript runs.
A robots.txt rule is a request, and bot-protection services can still challenge a crawler that robots.txt allows. Our homepage requests were plain HTTP requests, so a site that refused them may still let verified search crawlers through. The full results are available as a CSV.
Few AI products block AI search
Of the 168 sites with a readable robots.txt, 12 (7%) block at least one AI crawler, and most of them block training rather than search. Seven sites, including Canva, Lovart, Meshy and Muse, block OpenAI's training crawler while still allowing its search crawler: they opt out of training without leaving ChatGPT search.
Only 4 sites block an AI search crawler, and all four are AI assistants themselves: ChatGPT's site blocks PerplexityBot, Claude's blocks OpenAI's search crawler, and Meta AI and Poe block several. For a product that wants to be recommended, blocking AI search crawlers works against you.
The bigger risk: pages crawlers cannot read
20 of the 158 homepages we could load (13%) contain fewer than 500 characters of text before JavaScript runs, so a crawler that does not render JavaScript sees an almost empty page. Google renders JavaScript, but its own documentation notes that not all bots can, and recommends server-side rendering or prerendering.
Another 24 sites (13%) refused a plain automated request with an error or a challenge, whatever user agent we sent. An allow rule in robots.txt does not bypass bot protection, so confirm that the AI crawlers you care about are allowed, and that your pricing and docs pages are not behind the same wall.
llms.txt is common, but optional
101 of the 182 sites (55%) publish an /llms.txt file, a Markdown summary of the site with links to its key pages. First proposed in 2024, llms.txt is still a proposal; the current v2 permits files at the site root or under a path such as /docs/, and recommends links to clean Markdown versions of pages. Google says no such file is needed for its AI features.
It is cheap to add and useful for agents that read documentation, so it is worth publishing once your main pages are in order. Treat it as a guide to pages that already work, not a replacement for them.
Structured data describes the company more often than the product
98 of the 158 homepages we loaded (62%) include JSON-LD structured data. Most of it describes the organization and the website; only 48 (30%) describe the product itself as a SoftwareApplication, WebApplication or Product, with details such as category, platform and price.
Google uses structured data to understand page content. Keep it consistent with the visible text on the page; no special structured data is required for Google’s AI features.
A short checklist
Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot in robots.txt. If you do not want your content used for training, block the training crawlers separately.
Make sure your homepage, pricing page and docs show their key facts in the HTML that loads first. Let verified crawlers through bot protection using their published IP ranges; OpenAI, Anthropic and Perplexity all publish lists, and Perplexity recommends checking both the IP address and user agent.
State the facts an assistant needs in plain sentences: what the product does, who it is for, what it costs and its limits. Keep your product name and URLs stable, and redirect old ones after a rename.
Add product-level structured data and, optionally, an llms.txt that points to your best pages.
Sources & method
Official documentation was checked on 2026-10-14. Selection advice is our assessment. Interfaces and requirements may change; check the linked source for the version you intend to use.
- Dataset (CSV): crawler access and metadata for 182 AI product sites, checked October 2, 2026
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity crawlers
- Google's common crawlers (Google-Extended)
- Google: AI features and your website
- The /llms.txt proposal
- Google: JavaScript SEO basics
- Google: Introduction to structured data
How SOTA reviews projects · Suggest a correction · Download the evaluation checklist