Submit projectSubmit

THE SOTA FIELD GUIDE

How to get your AI product found by ChatGPT, Claude and Perplexity

Which AI crawlers to allow, what they need to read on your site, and what 182 AI product sites did in our October 2, 2026 snapshot.

By SOTA · Updated 2026-10-14 · Researched with AI assistance; facts checked against the linked sources · How we work

The short answer

AI assistants find products through search crawlers that respect robots.txt, through pages users ask them to open, and through ordinary search indexes. Let the search crawlers in, put your key facts in HTML that loads without JavaScript, and make pricing and docs easy to fetch. In our check of 182 AI product sites, only 4 blocked an AI search crawler, but 13% served homepages that are nearly empty without JavaScript and another 13% turned away a plain automated request.

Summary of the findings
CrawlerCompanyUsed forFollows robots.txtSites blocking it (of 168)
GPTBotOpenAITraining dataYes9
OAI-SearchBotOpenAIChatGPT search resultsYes3
ChatGPT-UserOpenAIPages a user asks ChatGPT to openMay not applyNot checked
ClaudeBotAnthropicTraining dataYes7
Claude-SearchBotAnthropicSearch quality in ClaudeYes2
Claude-UserAnthropicPages fetched in response to a user questionYesNot checked
PerplexityBotPerplexityPerplexity search resultsYes3
Perplexity-UserPerplexityPages fetched to answer a questionGenerally notNot checked
Google-ExtendedGoogleControl token for Gemini training and grounding, not a separate crawlerYes6

How AI assistants find products

AI companies use different bots for different jobs. OpenAI separates GPTBot training crawls from OAI-SearchBot search crawls; Anthropic separates ClaudeBot from Claude-SearchBot. PerplexityBot serves search and is not used to crawl content for AI foundation models. Training and search controls are separate: OpenAI says sites that block OAI-SearchBot are left out of ChatGPT search answers, though they may still appear as navigational links.

A third kind of request comes from users. OpenAI says robots.txt may not apply to ChatGPT-User requests. Perplexity says Perplexity-User generally ignores robots.txt. Anthropic says all its bots follow robots.txt, and disabling Claude-User prevents Claude from retrieving your content in response to a user query.

OpenAI says search access changes can take about 24 hours after a robots.txt update; Perplexity says its changes may take up to 24 hours. If GPTBot and OAI-SearchBot are both allowed, OpenAI may reuse one crawl for both purposes. Its OAI-AdsBot only visits pages submitted as ads.

Google's AI Overviews and AI Mode use the regular Search index. Google says there are no extra requirements and no special files or markup needed: a page that is indexed and eligible for a snippet can be used as a supporting link.

Google-Extended controls Gemini training and grounding through robots.txt and has no separate HTTP user agent. It does not affect inclusion or ranking in Google Search.

What we checked

On October 2, 2026 we checked the websites of 182 AI products in the SOTA catalog, counting each site once and leaving out products that live on a section of another company's site. For each site we read robots.txt and tested whether common AI crawlers may fetch the homepage, looked for /llms.txt and a sitemap, and fetched the homepage to record its title, description, share image, structured data and how much text it contains before JavaScript runs.

A robots.txt rule is a request, and bot-protection services can still challenge a crawler that robots.txt allows. Our homepage requests were plain HTTP requests, so a site that refused them may still let verified search crawlers through. The full results are available as a CSV.

The bigger risk: pages crawlers cannot read

20 of the 158 homepages we could load (13%) contain fewer than 500 characters of text before JavaScript runs, so a crawler that does not render JavaScript sees an almost empty page. Google renders JavaScript, but its own documentation notes that not all bots can, and recommends server-side rendering or prerendering.

Another 24 sites (13%) refused a plain automated request with an error or a challenge, whatever user agent we sent. An allow rule in robots.txt does not bypass bot protection, so confirm that the AI crawlers you care about are allowed, and that your pricing and docs pages are not behind the same wall.

llms.txt is common, but optional

101 of the 182 sites (55%) publish an /llms.txt file, a Markdown summary of the site with links to its key pages. First proposed in 2024, llms.txt is still a proposal; the current v2 permits files at the site root or under a path such as /docs/, and recommends links to clean Markdown versions of pages. Google says no such file is needed for its AI features.

It is cheap to add and useful for agents that read documentation, so it is worth publishing once your main pages are in order. Treat it as a guide to pages that already work, not a replacement for them.

Structured data describes the company more often than the product

98 of the 158 homepages we loaded (62%) include JSON-LD structured data. Most of it describes the organization and the website; only 48 (30%) describe the product itself as a SoftwareApplication, WebApplication or Product, with details such as category, platform and price.

Google uses structured data to understand page content. Keep it consistent with the visible text on the page; no special structured data is required for Google’s AI features.

A short checklist

Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot in robots.txt. If you do not want your content used for training, block the training crawlers separately.

Make sure your homepage, pricing page and docs show their key facts in the HTML that loads first. Let verified crawlers through bot protection using their published IP ranges; OpenAI, Anthropic and Perplexity all publish lists, and Perplexity recommends checking both the IP address and user agent.

State the facts an assistant needs in plain sentences: what the product does, who it is for, what it costs and its limits. Keep your product name and URLs stable, and redirect old ones after a rename.

Add product-level structured data and, optionally, an llms.txt that points to your best pages.

Sources & method

Official documentation was checked on 2026-10-14. Selection advice is our assessment. Interfaces and requirements may change; check the linked source for the version you intend to use.

How SOTA reviews projects · Suggest a correction · Download the evaluation checklist

Explore the projects