Why the same question gets a different answer

The first time most people run an AI visibility check, they do the obvious thing: they open ChatGPT, type the same question, and compare. The brands are different. Then they ask again, and the brands change again.

It is tempting to decide that either the tool or the assistant is broken. Neither is. AI answers are generated, not looked up, and several things feed into each one. Once you know what they are, the variation stops being alarming and starts being something you can plan for.

Two ways an assistant can answer

The biggest switch is whether the assistant searches the web before it answers.

The crawler documentation makes the split visible. OpenAI separates GPTBot, used for training models, from OAI-SearchBot, which surfaces sites in ChatGPT’s search answers. Anthropic lists ClaudeBot for training alongside Claude-SearchBot and Claude-User for search and user requests. Perplexity describes PerplexityBot as indexing sites for its search results and says it “is not used to crawl content for AI foundation models.”

For measurement this matters a lot. A tool that asks a model without search can report your brand as missing, while the version real people use, with search switched on, names you. Ask any vendor which mode each platform is collected in.

Six things that move an answer

The third one surprises people. Ahrefs describes fan-out queries as “extra questions that AI platforms generate when answering a user’s prompt”. The reader never sees them, but they decide which pages the assistant reads. Two slightly different questions can fan out into quite different searches.

What good tools do about it

1. Fix the conditionsSame wording, platform, country and language every run.
2. Repeat the questionSeveral runs, so one odd answer doesn’t swing the result.
3. Show the sampleRates with a count, such as “12 of 30 answers”.
4. Keep the textSaved answers let you check what actually changed.

None of this removes variation. Our guide to reading AI visibility results covers what to look for in the saved answers. It makes the variation visible, so you can tell a real shift from a coin toss. Our piece on reading week-to-week movement shows how to do that with a simple rule.

Checking an answer yourself

Spot checks in your own browser are still worth doing, as long as you read them for what they are.

A note on Google

Google’s AI Overviews and AI Mode are built on Google Search itself. Google’s own guidance says there are “no additional requirements to appear in AI Overviews or AI Mode”, and that their traffic is counted inside the normal Search Console performance report. If Google is your main concern, your existing search data is part of the picture. AI visibility tools add the answer text and the competitor view on top.