By Hridoy Ahmed | Last updated: October 2026 | 8 min read
Short answer: Companies share only part of how their AI tools pick sources. Google says its AI features find pages in its Search index and use related searches to build answers. OpenAI, Anthropic and Perplexity publish which crawlers they use. Nobody outside these companies knows exactly how pages are ranked inside an AI answer, so be careful with anyone who claims to.
In this guide
- Google: what is documented
- ChatGPT, Claude and Perplexity: what is documented
- What is not public
- What to do anyway
- FAQ
Google: what is documented
Google’s guide, Optimizing your website for generative AI features on Google Search, explains the basics:
- The AI features rely on Google’s core Search ranking and quality systems.
- They find relevant, up-to-date pages in the Search index and read them to write an answer, with links to supporting pages.
- They may run several related searches at once, called query fan-out, to get more information.
- A page must be indexed and eligible to show a snippet.
Google also says AI Overviews and AI Mode may use different models and techniques, so the pages shown can differ. See my guide to Google AI Overviews and AI Mode.
ChatGPT, Claude and Perplexity: what is documented
These companies mostly publish information about their crawlers, not about how answers are ranked.
- OpenAI: says OAI-SearchBot is used to surface websites in ChatGPT’s search features, and that sites opted out of it will not be shown in ChatGPT search answers. GPTBot is a separate crawler for model training.
- Anthropic: says Claude-SearchBot helps improve search result quality and Claude-User may fetch pages when a person asks Claude a question.
- Perplexity: publishes crawlers for search indexing and for user-requested fetches.
The details, and how to allow or block each bot, are in my guide to AI crawlers and robots.txt. Allowing a crawler makes a page reachable. It does not mean the page will be chosen.
What is not public
- The exact ranking or scoring used to pick sources inside an answer
- How much weight each tool gives to freshness, links, brand mentions or page structure
- Whether a tool uses its own index, a partner’s search results or a mix, and how that changes
- Why a source is cited in one answer and not in the next
Studies and tools that claim to explain these usually test a limited set of questions on a given day. Results can change when the tool changes. Treat them as clues, not rules. This is also why I treat things like llms.txt as unproven. See what llms.txt is and whether you need it.
What to do anyway
You do not need the secret formula to do the useful things:
- Be crawlable and indexed. This is the entry ticket for every search system.
- Create original, helpful content with first-hand knowledge. Google says this is likely to matter most in the long run.
- Answer questions clearly, with the answer near the top. See how to structure content so AI assistants can cite it.
- Make your brand clear and consistent. See entity SEO.
- Choose your crawler access on purpose.
- Measure with care, using Search Console, analytics and spot checks. See how to measure AI search visibility.
If you would like help with these steps, see my AI search optimization service.
Common mistakes
- Believing a claim about how AI “ranks” pages without evidence
- Copying tactics from one tool and assuming they work in all tools
- Paying for fake mentions or special files promised to boost AI visibility
- Judging success from a single answer
- Forgetting that tools and rules change
FAQ
How does ChatGPT choose which sites to cite?
OpenAI has not published a full ranking method. It documents that OAI-SearchBot supports ChatGPT search, and that sites opted out of it will not be shown in ChatGPT search answers.
Does Google use different rules for AI Overviews?
Google says its AI features are rooted in its core Search ranking and quality systems, and that a page must be indexed and eligible to show a snippet.
Can I find out why my page was cited?
Not reliably. Google’s Search Console shows impressions in its AI features, but not the reason. Other tools do not provide this.
Will this change?
Very likely. I update my AI search guides when official documentation changes.
Want advice that separates facts from guesses? That is how I work. See my AI search optimization service.
About the author: Hridoy Ahmed is an SEO specialist and WordPress security expert based in Dhaka, Bangladesh. He has 8+ years of experience and works with clients worldwide.