AI search visibility: crawler controls, llms.txt and quotable pages

The CIQRA team30 September 2026 8 min read

To be found and cited in AI answers, a page first has to be crawlable, indexed and quotable by the system producing the answer. For Google's AI Overviews and AI Mode, Google says there are no extra requirements: a page must be indexed and eligible to appear in Search with a snippet. For ChatGPT search and Perplexity, you must allow their search crawlers (OAI-SearchBot and PerplexityBot) in robots.txt. An llms.txt file is an optional, useful extra for AI agents, but no search engine has said it uses it for ranking or citations.

Below: what the vendors document, then a checklist. For how AI summaries affect organic traffic, seethe importance of SEO in driving organic traffic.

How AI answers choose their sources

The old question ("can this engine find and read my page?") still comes first. The new one is whether a passage on your page is clear enough to support an answer.

Google AI Overviews and AI Mode

Google's Search Central page on AI features (last updated December 2025) is direct: "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary." To be shown as a supporting link, a page must be "indexed and eligible to be shown in Google Search with a snippet."

Google also explains that both features may use a "query fan-out" technique, issuing multiple related searches across subtopics to build a response, so pages that answer specific sub-questions well matter. Traffic from AI features is counted in the Search Console Performance report under the "Web" search type.

Google lists the fundamentals that still matter:

  • crawling allowed in robots.txt, and by any CDN or hosting infrastructure;
  • content findable through internal links;
  • a good page experience (see our guide to Core Web Vitals for stores);
  • important content available as text, supported by good images and video where relevant;
  • structured data that matches the visible text on the page;
  • Merchant Center and Business Profile information kept up to date.

And one sentence that settles a lot of debate: "You don't need to create new machine readable files, AI text files, or markup to appear in these features. There's also no special schema.org structured data that you need to add."

ChatGPT search and Perplexity

OpenAI states that OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features," and recommends allowing it if you want to appear there. Perplexity says PerplexityBot is "designed to surface and link websites in search results on Perplexity." In both cases, blocking the search crawler is the most reliable way to stay out of those answers, and allowing it is the entry ticket.

Crawler controls: one bot per purpose

Google, OpenAI and Perplexity each document separate tokens for search and for other uses, such as model training, so you can say yes to one and no to the other.

VendorTokenWhat it is for, per the vendor
GoogleGooglebotCrawling for Google Search, including AI Overviews and AI Mode. Google says robots.txt rules for Googlebot are the control for how sites are crawled for Search.
GoogleGoogle-ExtendedA product token (not a separate crawler) that controls whether crawled content may be used to train future Gemini models and for grounding in Gemini apps. Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."
OpenAIOAI-SearchBotSurfacing sites in ChatGPT search results.
OpenAIGPTBotCrawling content that may be used to train OpenAI's foundation models. Disallowing it signals that your content should not be used for training.
OpenAIChatGPT-UserVisits triggered by a user's action in ChatGPT. OpenAI says robots.txt rules may not apply, and it is not used to decide what appears in search.
PerplexityPerplexityBotSurfacing and linking sites in Perplexity search results; Perplexity says it is not used to crawl content for AI foundation models.
PerplexityPerplexity-UserFetching a page when a user asks a question; Perplexity says it generally ignores robots.txt because the request is user-initiated.

OpenAI spells out that "each setting is independent of the others": you can allow OAI-SearchBot to appear in search while disallowing GPTBot. It also notes that it can take about 24 hours for its systems to pick up a robots.txt change.

A worked example

A store that wants to be cited in AI search but not used for model training could use these robots.txt groups:

  • User-agent: Googlebot, Allow: / (Search, including AI Overviews and AI Mode)
  • User-agent: Google-Extended, Disallow: / (Gemini training and grounding)
  • User-agent: OAI-SearchBot, Allow: / (ChatGPT search)
  • User-agent: GPTBot, Disallow: / (OpenAI model training)
  • User-agent: PerplexityBot, Allow: / (Perplexity search)

Two cautions. First, blocking Googlebot to keep content out of AI Overviews would also remove you from ordinary Google results; Google points to nosnippet, data-nosnippet, max-snippet and noindex if you want to limit what is shown. Second, Google explicitly says crawling must also be allowed by your CDN or hosting infrastructure, so check bot-protection rules at the edge too.

What llms.txt is, and what it is not

llms.txt is a proposal by Jeremy Howard, first published in September 2024 and revised as v2 in 2026 at llmstxt.org. It suggests a Markdown file at /llms.txt (or at a subpath such as /docs/llms.txt) that gives AI agents a short, curated guide to a site. The format is simple:

  1. an H1 with the name of the site or project (the only required part);
  2. a blockquote with a short summary;
  3. optional paragraphs or lists explaining how to read the rest;
  4. H2 sections containing lists of links, each with optional notes; a section called "Optional" holds links an agent can skip.

The proposal also recommends clean Markdown versions of important pages (for example page.html.md) and says llms.txt is used on demand, "when an agent needs information about a topic while assisting a user," mainly for inference rather than training. It is not a replacement for robots.txt, which controls access, or for your XML sitemap, which lists every indexable page.

What it is not: a ranking or citation signal that any search engine has confirmed. Google's AI features documentation says you do not need "AI text files" to appear in AI Overviews or AI Mode. We found no statement from OpenAI or Perplexity in their crawler documentation saying their search products read llms.txt. The one first-party mention we found is from Chrome: Lighthouse's agentic browsing audits check for the file, flag a server error when fetching it, and mark a missing file "Not Applicable" because "providing the file is optional at the moment."

Treat llms.txt as cheap housekeeping for agents that use it, not as a route into AI answers.

Make your content easy to quote

  • Answer first. Put the direct answer in the first two or three sentences under a heading.
  • One question per section. Headings that match how people ask help related searches land on the right passage.
  • Keep facts in text. Prices, dimensions, return windows and shipping terms belong in HTML text, not only in images.
  • Be specific and current. Dated, sourced facts are easier to cite than vague claims.

Structured data

Structured data is, in Google's words, "a standardized format for providing information about a page and classifying the page content." There is no special AI schema, and markup must match the visible text. Use types that describe your pages honestly:Organization, Product with offers, shipping and return details, BreadcrumbList, and FAQPage or Article where they fit.

Server-rendered HTML

Google's JavaScript SEO guide says server-side or pre-rendering "is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript." View the page source (not the inspector) and confirm your headings, copy and prices are there. When you change platforms, see ourreplatforming SEO checklist.

A practical checklist

  1. Open your robots.txt and list every AI-related token. Decide per purpose: search (Googlebot, OAI-SearchBot, PerplexityBot) versus training (Google-Extended, GPTBot).
  2. Check your CDN, firewall and bot-protection rules so the crawlers you allow are not blocked or challenged.
  3. Confirm key pages are indexed, not accidentally nosnippet, and have their main content in the HTML source.
  4. Validate structured data against what the page shows.
  5. Rewrite the opening of your most important pages so the answer comes first.
  6. Publish a short llms.txt that links to your key pages, and keep it in sync.
  7. Track the Search Console Performance report ("Web" search type) and AI referral traffic over time.

How CIQRA handles this

CIQRA publishes llms.txt and llms-full.txt for every site automatically, and you can edit them. You can set separate rules by purpose, for example blocking model training while staying open to answer engines, and edit the rules for search engines and AI crawlers in robots.txt without touching code. Pages are served as server-rendered HTML, so bots read your content without waiting for JavaScript, and sites ship with structured data (JSON-LD) for your organization, pages, FAQs and blog, or for products, prices, shipping and returns.

FAQ

Do I need llms.txt to appear in Google AI Overviews?

No. Google says you do not need new machine-readable files, AI text files or special markup to appear in AI Overviews or AI Mode. A page must be indexed and eligible to show with a snippet.

If I block GPTBot, will I disappear from ChatGPT search?

Not according to OpenAI. GPTBot and OAI-SearchBot are controlled independently; ChatGPT search relies on OAI-SearchBot.

Does blocking Google-Extended hurt my Google rankings?

Google says Google-Extended does not affect inclusion in Google Search and is not used as a ranking signal.

Sources

Share LinkedIn X Facebook Email

Written by
The CIQRA team

Related articles