Optimise Content for AI Retrieval Pipelines

Microsoft Copilot cited GoGoChimp 3,919 times for "optimize content for AI retrieval pipelines" in the 90 days to 25 September 2026, yet that was only a 10.04% share.

It was our top grounding query out of 598 in Bing's sample, and about nine citations in ten still went to other sites. If you want AI search to quote your pages, you're trying to close the same gap.

We all write pages as if the AI puts the kettle on and reads from the top. In practice it lifts a sentence or two and legs it. The wider playbook is in our guide to optimising for AI search engines and the AI SEO hub.

Bing Webmaster Tools shows the phrases Copilot searched before citing your pages. Bing calls them grounding queries, "the key phrases the AI used when retrieving content" (Bing Webmaster Blog, 2026). The list is the closest any engine comes to publishing its sub-queries. Citation share is your cut of all the citations Bing showed for one phrase.

The report covers "Microsoft Copilot, AI-generated summaries in Bing, and select partner integrations", and the data "represents a sample of overall citation activity" (Bing Webmaster Blog, 2026). It says nothing about ChatGPT, Perplexity or Google. Microsoft calls citation share observational rather than a ranking, and it doesn't show who holds the rest (Bing Search Blog, 2026).

In those 90 days, Bing logged 44,677 Copilot citations of gogochimp.com, part of more than 47,000 since April 2026. Its query report named 598 grounding queries covering 32,713 of them. Here are the retrieval and grounding rows, with two of our stronger queries for contrast.

Grounding query (Bing's wording)	Copilot citations of GoGoChimp	GoGoChimp's citation share
optimize content for AI retrieval pipelines	3,919	10.04%
improving content grounding in AI search	1,466	11.79%
passage retrieval answer engine optimization relationship	440	3.88%
how to get content indexed by AI search engines	358	2.53%
strategies improving content grounding AI search	61	2.39%
optimize content for AI inference outcomes	60	0.70%
optimize content for AI agent retrieval solutions	30	1.41%
best A/B testing tools 2026 (contrast)	584	25.10%
UK conversion rate optimisation specialists SaaS Salesforce HubSpot (contrast)	46	73.02%

Source: GoGoChimp's Bing Webmaster Tools export of 28 September 2026, covering the 90 days to 25 September. Bing's data is a sample.

Because the topic is crowded. If 3,919 citations are 10.04% of the total, Bing's sample showed roughly 39,000 citations across all sites for that one phrase. That's my arithmetic, because Bing doesn't publish the total.

My guess, from one site's sample, is that a generic how-to query has plenty of interchangeable passages, so Copilot spreads its citations thin. A narrow query with few qualified sources concentrates them on whoever answers it precisely.

This page wasn't live then. Bing's query-to-page view shows all 3,919 went to one page, our guide to optimising for AI search engines, and every one landed between 11 and 28 August 2026, with none from then to 6 October. Treat that 10% as a three-week snapshot, not a steady share.

In Bing's sampled data for the 90 days to 25 September 2026, Copilot cited GoGoChimp 46 times for a narrow UK CRO specialist query at a 73.02% share, and 60 times for "optimize content for AI inference outcomes" at 0.70%.

What is an AI retrieval pipeline?

An AI search engine runs seven steps between your question and its cited answer. It decides whether to search, rewrites the question into sub-queries, retrieves candidates, fetches each page, splits it into passages, reranks them, then grounds the answer and cites the sources. No engine documents every step, and none publishes the weights that decide which source wins.

Most AI-search advice is about the last step, how a passage reads. The controlled tests say the rewrite, retrieval and rerank steps matter more.

Only ChatGPT's rewrite step has been measured. Ahrefs found ChatGPT ran 1.78 searches per prompt across 118,931 fan-out queries (Ahrefs, 2025). Google publishes no count for AI Mode, so ignore anyone quoting one. It also warns that pages built for every fan-out variant fall under its scaled content abuse policy (Google Search Central, 2026).

What do retrieval and reranking do to your page?

Retrieval builds the longlist, and reranking runs the interview. Perplexity has published the most detail. It retrieves with lexical and semantic methods, splits documents into "self-contained spans, each of which can be individually retrieved and ranked", and uses "cross-encoder reranker models" for the final cut (Perplexity Research, 2025).

Which index does each AI engine retrieve from?

Copilot retrieves from Bing's index, Google's AI features from Google's and Perplexity from its own. ChatGPT uses third-party search providers plus OpenAI's own index, and Anthropic lists Brave Search behind Claude. A page missing from the index an engine searches can't be retrieved, no matter how well it's written.

Engine	Index it retrieves from	Crawler to allow	Renders JavaScript?	Sources
Microsoft Copilot	Bing's index, from the same crawl as Bing search	bingbot (Microsoft lists no separate Copilot crawler)	Bingbot renders with Microsoft Edge. Copilot read raw HTML at answer time in a June 2026 test	Bing Webmaster Blog, 2025, Bing crawler list
ChatGPT search	Third-party search providers, plus OpenAI's own cached index	OAI-SearchBot	No, in 2024 crawler logs and the June 2026 test	OpenAI Help Center, 2026, OpenAI web search docs
Google AI Overviews and AI Mode	Google's index, for pages "eligible to be shown in Google Search with a snippet"	Googlebot	Yes. Google processes JavaScript unless it's blocked	Google Search Central, 2025, Google Search Central, 2026
Perplexity	Its own index, tracking "over 200 billion unique URLs"	PerplexityBot	No, in 2024 crawler logs and the June 2026 test	Perplexity Research, 2025
Claude	Brave Search, listed as Anthropic's web search subprocessor	Claude-SearchBot and Claude-User	No. Anthropic's web fetch tool doesn't support JavaScript-rendered sites	Anthropic Trust Center, Anthropic, web fetch tool

JavaScript column sources: Vercel, 2024, Alpar, 2026.

Where does ChatGPT get its search results?

From search providers OpenAI doesn't name, plus its own cache. Ranking on Google tells you little about getting cited by ChatGPT. Ahrefs found only 6.82% of ChatGPT's search results in Google's top 10 for the same fan-out queries (Ahrefs, 2025). Across ChatGPT, Gemini and Copilot, 12% of cited URLs ranked in Google's top 10 for the original prompt, and 80% didn't rank anywhere for it (Ahrefs, 2025).

Why does Brave matter for Claude?

Block Googlebot and you cut off the Brave layer behind Claude as well. Brave won't crawl a page Googlebot is blocked from (Brave Search Help), and Anthropic's Trust Center lists Brave Search as a subprocessor for web search (Anthropic Trust Center). Simon Willison spotted the listing in March 2025 and matched Claude's citations to Brave's results (Willison, 2025).

Which crawlers should you allow, and do they render JavaScript?

Allow each engine's search crawler and decide on training crawlers separately. OpenAI and Anthropic each run separate bots for search, training and user-triggered fetches, and you can block one without the others. On JavaScript, assume the major US assistants read only your raw HTML when fetching a page for an answer.

Company	Search index crawler	Training crawler	User-triggered fetcher	What the documentation says
OpenAI	OAI-SearchBot	GPTBot	ChatGPT-User	Block OAI-SearchBot and you "will not be shown in ChatGPT search answers". For ChatGPT-User, "robots.txt rules may not apply"
Anthropic	Claude-SearchBot	ClaudeBot	Claude-User	All three honour robots.txt. Blocking Claude-User "may reduce your site's visibility for user-directed web search"
Perplexity	PerplexityBot	None. PerplexityBot "is not used to crawl content for AI foundation models"	Perplexity-User	Perplexity-User "generally ignores robots.txt rules"
Google	Googlebot	Google-Extended, a robots.txt token rather than a crawler	Not applicable	The AI Overviews and AI Mode opt-out is a Search Console setting
Microsoft	bingbot	None listed separately	None listed separately	One crawler serves Bing search and Copilot

Sources: OpenAI crawler docs, Anthropic Help Center, 2026, Perplexity crawler docs, Google's common crawlers, Bing crawler list.

In our October 2026 AI citability benchmark, 16 of 299 sites blocked a search or answer-time AI bot in robots.txt, and 12 more blocked training crawlers only.

Does Google-Extended keep you out of AI Overviews?

No. Google says Google-Extended "does not impact a site's inclusion in Google Search". It governs Gemini training and grounding in Gemini Apps and Vertex AI (Google's common crawlers).

The control for AI Overviews and AI Mode is the "Search generative AI" setting in Search Console. It's been open to every site worldwide since 31 August 2026 and isn't a ranking signal elsewhere in Search (Search Console Help, 2026).

What happens to answers that only appear after JavaScript runs?

The major US assistants don't see them. Vercel and MERJ's December 2024 log study found "none of the major AI crawlers currently render JavaScript", naming OpenAI's crawlers, ClaudeBot and PerplexityBot (Vercel, 2024).

In June 2026, Andre Alpar gave twelve assistants a page whose real reference number appeared only after JavaScript ran. ChatGPT, Claude, Gemini, Perplexity, Meta AI and Copilot all reported the decoy from the raw HTML (Alpar, 2026). Microsoft adds that content hidden in tabs or expandable menus "may not render" (Microsoft Advertising, 2025).

On 9 October 2026 we fetched three of our own AI-search guides with an OAI-SearchBot user agent, and all three returned their full text in the first HTML response. Webflow renders on the server, so I can't take the credit. If OpenAI's crawlers start running scripts, this advice changes for ChatGPT first.

Do AI engines read whole pages or lift passages?

They lift passages, and the quoted unit is tiny. Perplexity ranks "self-contained spans" (Perplexity Research, 2025), and Microsoft's grounding tool for agents returns "relevant chunks" (Microsoft Learn, 2026). Each Claude citation carries "Up to 150 characters of the cited content" (Anthropic, web search tool).

Most of a long page goes unused. DEJAN AI studied 7,060 grounding queries through Gemini's API. Pages under 1,000 words had 61% of their text used, and pages over 3,000 words 13% (DEJAN AI, 2025). This guide runs past 3,000 words, so every section has to make sense when it's lifted out on its own.

Should you break your content into chunks for AI?

No, but don't write walls of text either. Google says "There's no requirement to break your content into tiny pieces" (Google Search Central, 2026). Microsoft's Krishna Madhavan tells publishers to "Avoid long walls of text" and "Make answers snippable" with lists, Q&As and tables (Microsoft Advertising, 2025).

I keep both happy with one question per section, answered straight away with its evidence. Then I paste each section into a blank document and read it cold. If it leans on "this approach" or a bare "it", I rewrite the sentence.

Anthropic measured the cost of lost context in a lab benchmark on private documents. Prefixing each chunk with 50-100 tokens of document context cut top-20 retrieval failures from 5.7% to 3.7%, and to 1.9% with contextual BM25 and reranking added (Anthropic, 2024). Claude's web search doesn't work like this, but the test shows why a section needs its own context.

Is there an ideal passage length for AI citation?

No primary source backs one. The popular 120-180 word rule comes from SE Ranking's ChatGPT study of 216,524 pages. Sections of that length averaged 4.6 citations and sections under 50 words 2.7, but sections over 180 words did better again at 5.7 (SE Ranking, 2025). In SE Ranking's AI Mode study, sections of 35 words or fewer averaged 4.3 citations and 100-150 word sections 4.7 (SE Ranking, 2025).

Sections under 50 words do worse, and that's all the data supports. Our 40-60 word answer under each question heading is house style. Anyone selling you a word count as an engine spec is guessing.

What decides which passage gets cited?

Relevance to the specific sub-query, which shows up as retrieval rank, is the strongest predictor measured so far. In AirOps data analysed by Kevin Indig, ChatGPT cited 58% of pages in retrieval position 0 and 14% at position 10 (Indig, 2026). Sprinklr's controlled trials agree. They named topical relevance and list position as "the biggest drivers of being cited first" (Sprinklr, 2026).

Why does ChatGPT skip pages it has already retrieved?

Ahrefs' analysis suggests ChatGPT judges the title, URL and snippet before opening the page. Its study of 1.4 million prompts found ChatGPT cites about half the URLs it retrieves (Ahrefs, 2026). Cited pages had titles closer to the prompt, with an average cosine similarity of 0.602 against 0.484. Results with natural-language URL slugs were cited 89.78% of the time, against 81.11% without.

In the same AirOps data, pages whose headings closely matched the query were cited 41% of the time, and weak matches 30% (Indig, 2026). The always-cited and never-cited pages looked alike on length, at about 2,200 words each. These are correlations, but they all point at the same fields. I write titles, slugs and headings to match the sub-query and stop fretting about length.

Where on the page should the answer go?

In the first third of the text. Kevin Indig matched 18,012 verified ChatGPT citations to their source sentences and found 44.2% came from the first 30% of the page, against 24.7% from the final third (Indig, 2026). That's why our own data sits near the top of this guide. Screen position doesn't seem to count. SALT.agency measured depth in screen pixels across 2,318 AI Mode citations and found no correlation (SALT.agency, 2025).

Indig's study also found heavily cited text averaged 20.6% proper nouns, against a typical 5-8% in English (Indig, 2026). Our entity SEO guide covers the naming side.

Which popular tactics fail controlled tests?

Schema markup and the 2023 GEO rewrites both failed controlled tests, and word-count targets, FAQ schema and llms.txt show no benefit in the large datasets. A pattern among cited pages doesn't prove an edit causes citations, so the table grades each claim by the kind of study behind it.

Tactic or factor	Best evidence	Type of study	What it found
Structural fields (titles, headings, descriptions, schema)	SAGEO, 2026	Controlled, simulated search pipeline	Optimising only those fields raised top-20 retrieval by 22% and citation by 2%
Rewriting body text, GEO style	SAGEO, 2026	Controlled, simulated search pipeline	Cut top-20 retrieval by 9% and citation by 6%
Adding schema (JSON-LD)	Ahrefs, 2026	Controlled, live engines	No measurable lift on ChatGPT or AI Mode. AI Overview citations fell 4.6%
Adding quotations, statistics and sources	Bajemon and Rochet, 2026	Controlled proxy, ten model families	Moved citation on none of them
Hitting a word count	SE Ranking, 2025, Ahrefs, 2025	Correlational	No optimum. Word count correlated with AI Overview citation at 0.04
FAQ schema	SE Ranking, 2025	Correlational	3.6 ChatGPT citations with it, 4.2 without
llms.txt	Google Search Central, 2026	Vendor statement	Google Search doesn't use the files
Fresh, accurate dates	Fang et al., 2025, Ahrefs, 2025	Lab, plus correlational	Rerankers favour recent dates. AI-assistant citations were 25.7% "fresher" than organic results

Schema is the clearest case of correlation fooling people. The same Ahrefs study found cited pages across 6 million URLs were almost three times more likely to carry JSON-LD. Yet adding it to 1,885 pages, measured against matched controls, produced no measurable lift on any engine (Ahrefs, 2026). Keep schema accurate and matched to the visible text, for eligibility and rich results.

Do the 2023 GEO tactics still work?

Not on current models, as far as anyone has tested. The 2023 GEO benchmark, which coined generative engine optimisation, fed the top five Google results to gpt-3.5-turbo. It measured how much of each answer a source took up and never checked whether it was retrieved or cited (Aggarwal et al., 2024). Its top three methods gave a "relative improvement of 30-40%", and quotations alone 41%.

In September 2026, Elisha Bajemon and André-Louis Rochet of TW3 Partners (Citead) re-ran the three levers as paired, length-controlled edits. The levers "moved citation on none of ten engine families" (Bajemon and Rochet, 2026). The only clear effect, from adding cited sources, was a fall of 0.79 points in citation share.

The ten families were models called through APIs, and the authors call their setup "a proxy, not open-web competition with retrieval and deduplication". Their company sells a page citability score, and the paper carries no conflict-of-interest statement. C-SEO Bench, an independent NeurIPS 2025 benchmark, found published rewriting methods "largely ineffective", with statistically significant gains in 3 of 54 cases (Puerto et al., 2025).

Bajemon and Rochet also found that a query-blind content score barely tracked citation share, at a correlation of 0.114. A query-aware relevance model reached 0.388 (Bajemon and Rochet, 2026). The authors advise treating such scores "as quality filters rather than citation predictors". That goes for every page score on the market, ours included.

I still put the source beside the number. Readers can check it, and Microsoft says examples, data and cited sources help build trust (Bing Webmaster Blog, 2026).

What is content grounding in AI search, and how do you improve it?

You improve grounding with discrete, dated facts that name their sources and agree with each other across the page. Google defines grounding as "the ability to connect model output to verifiable sources of information" (Google Cloud, 2026).

Microsoft says the unit of value is now "discrete, supportable facts with clear provenance", and that "a stale fact produces a misleading response" (Bing Search Blog, 2026). Two different figures for the same thing give Copilot a reason to leave you out. Our guide to improving content grounding in AI search has the full method.

How does passage retrieval relate to answer engine optimisation?

Answer engine optimisation is how you write for passage retrieval. A question heading with a self-contained answer under it is a passage built to match one sub-query and be quoted without the rest of the page.

Google began ranking individual passages in 2020, saying it would "improve 7 percent of search queries across all languages" (Google, 2020). AI answers go further and quote the passage itself. Our guide to writing an answer-first opening covers the first lines of each section.

How do you get content indexed by AI search engines?

There's no AI index to submit to. You get into the indexes the engines already search and let their crawlers in. That means Bing for Copilot, Google for AI Overviews and AI Mode, and Brave for Claude. For ChatGPT and Perplexity, allow the search crawlers in the table above.

Bing revisits submitted sitemaps "typically at least once per day" (Bing Webmaster Blog, 2025). IndexNow is the doorbell for changes. A ping is shared with every participating engine, though Google isn't one, and it "does not guarantee immediate indexing" (IndexNow). Our Copilot SEO guide goes deeper on Bing.

How do you measure AI retrieval?

Measure with first-party data and treat third-party trackers and vendor scores as directional. The six free sources below cover Copilot, Google, referral visits and crawler activity, and none of them reports ChatGPT citations.

Source	What it shows	What it doesn't show	Available since
Bing Webmaster Tools AI Performance	Copilot and Bing AI citations, cited pages, grounding queries and citation share, from a sample	ChatGPT, Google or Perplexity activity	Public preview 10 February 2026. Citation share added June 2026
Search Console generative AI report	Impressions in AI Overviews, AI Mode and Discover, by page, country, device and date	Clicks, click-through rate, position or queries	3 June 2026. Every site worldwide since 31 August 2026
GA4 AI Assistant channel	Visits from recognised AI assistants, with the medium "ai-assistant"	Clicks from AI Overviews and AI Mode, which stay in Organic Search	13 May 2026
`utm_source=chatgpt.com`	ChatGPT referral visits, tagged automatically by OpenAI	Citations that didn't earn a click	Documented by OpenAI
Microsoft Clarity Citations	Page citations, grounding queries and a "share of authority" metric	ChatGPT citations	Generally available 13 May 2026
Server logs	Which AI crawlers fetched which pages, and how many requests were 404s	Whether anything was cited	Always, if you check bot IPs against each vendor's published list

Sources: Bing Webmaster Blog, 2026, Bing Search Blog, 2026, Google Search Central Blog, 2026, Google Analytics release notes, GA4 default channel group, OpenAI Publishers FAQ, Microsoft Clarity, 2026, OpenAI crawler docs.

Use the grounding queries as keyword research. Each month, sort them by citations, flag the high-volume ones where your share is low and give each a section whose heading sits close to its wording. That's how I chose this guide's headings.

Bing's dashboard also shows which pages are cited for each grounding query (Microsoft Advertising, 2026). For this guide's main query, it showed every citation going to a single older guide of ours, all inside three weeks of August. Our AI SEO measurement stack walks through the setup, and our comparison of free AI SEO monitoring tools covers the third-party trackers.

None of these tells you why a page isn't quoted. Rubric, the citability auditor we build at GoGoChimp, checks headings, answer-first openers and whether schema survives without JavaScript. It predicts citability and doesn't count citations, so use Bing's data to see whether a fix worked.

Why do one-off prompt checks mislead?

AI answers change between runs. In replayed agentic-search transcripts, 15% of binary citation decisions flipped when an answer was regenerated (Selvam and Ghosh, 2026). Only 18% of the web pages AI Overviews drew on were common between two runs of the same query (Kirsten et al., 2026). Read monthly trends across many queries, and ignore screenshots of single prompts.

Questions about AI retrieval pipelines

Does blocking GPTBot remove my site from ChatGPT search?

No. GPTBot is OpenAI's training crawler. OAI-SearchBot is the one that matters for ChatGPT search answers, and ChatGPT-User makes the live fetch when a question sends ChatGPT to your page (OpenAI crawler docs). GPTBot and OAI-SearchBot are set separately in robots.txt. Block OAI-SearchBot and your pages can still appear as links, but not in search answers.

How do you test whether ChatGPT can read your page?

Give ChatGPT one page URL and ask for a detail only that page carries. If it quotes the detail, the live fetch got through. A header checker set to "GPTBot" tests the wrong bot. A spoofed user agent can't prove or disprove a block either, because firewall rules for verified bots only fire on the vendor's real IP addresses. Our ChatGPT SEO guide has the full curl test.

Does llms.txt help AI search?

Not on current evidence. Google says its Search systems don't use llms.txt files (Google Search Central, 2026), and SE Ranking found the file had negligible impact on ChatGPT citations (SE Ranking, 2025). Our llms.txt explainer covers what the file is for.

How long until changes show up in AI answers?

Only crawler controls have documented timings. OpenAI's search crawler takes about 24 hours to pick up a robots.txt change, Perplexity's up to 24 hours, and Google's AI opt-out one to two days for most content (OpenAI, Perplexity, Google). Content changes wait for recrawling, which Google says can take days to months (Google Search Central, 2025).

How do you optimise content for AI agents rather than AI search?

Much the same way, because agent tools retrieve passages too. Microsoft's Grounding with Bing Search tool returns "relevant chunks" of web pages (Microsoft Learn, 2026), and Anthropic says comparative research can use 10 or more searches (Anthropic, web search tool). More searches mean more sub-queries for each section to match.

Can you optimise content for AI inference outcomes?

Only indirectly. Inference is where the model writes the answer, and you have no say in that. Your influence stops at what it can retrieve and whether your facts support a claim. Microsoft says grounding can abstain when support is missing, stale or conflicting (Bing Search Blog, 2026), so keep your facts current and consistent.

What should you fix first to optimise content for AI retrieval pipelines?

Fix access first, because it's all or nothing. A blocked crawler or a JavaScript-only answer takes a page out of the running, and the tactics at the bottom of this list show no measurable lift. The list runs from the strongest evidence to the weakest.

Let every engine in. Allow OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot, bingbot and Googlebot, check your CDN or firewall isn't blocking them, and decide on GPTBot, ClaudeBot and Google-Extended separately. Evidence: vendor documentation.
Serve every answer in the first HTML response. Fetch your top pages with a search-bot user agent and compare the text with the rendered page. Evidence: a 2024 log study and a June 2026 answer-time test.
Check you're indexed where each engine looks, with Bing Webmaster Tools, Search Console, sitemaps and IndexNow. Evidence: vendor documentation.
Match titles, slugs and headings to the sub-questions, using Bing's grounding-query wording where you have it. Evidence: large correlational studies and one controlled pipeline test.
Put the main answer and its evidence in the first third of the page, and make each section stand alone. Evidence: correlational citation data and the engines' own passage-retrieval documentation.
Keep every fact sourced and current, and change the visible date only when a fact changes. Evidence: lab tests on rerankers and live freshness data.
Measure monthly with Bing grounding queries, Search Console AI impressions, GA4 and verified logs. They're first-party, so they outrank any vendor score, ours included.
Stop paying for schema as a citation lever, GEO-style rewrites, word-count targets and llms.txt. Evidence: no lift in controlled tests or in the large datasets.

This guide follows that checklist, and our next Bing exports will show whether it moves our 10.04%.

References

Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., & Deshpande, A. (2024). GEO: Generative Engine Optimization. KDD 2024. arXiv:2311.09735 (v3, 28 June 2024). https://arxiv.org/abs/2311.09735
Alpar, A. (2026, 9 June, updated 2 July). Do AI Assistants Actually Render Your JavaScript when Grounding? We Put It to the Test. Search Engine World. https://www.searchengineworld.com/do-ai-assistants-actually-render-your-javascript-when-grounding-we-put-it-to-the-test
Anthropic. (2024, 19 September). Introducing Contextual Retrieval. Anthropic Engineering. https://www.anthropic.com/engineering/contextual-retrieval
Anthropic. (2026). Web search tool (Claude Developer Platform documentation, read 9 October 2026). https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool
Anthropic. (2026). Web fetch tool (Claude Developer Platform documentation, read 9 October 2026). https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-fetch-tool
Anthropic. (2026). Subprocessors (Anthropic Trust Center, read 9 October 2026). https://trust.anthropic.com/subprocessors
Anthropic. (2026, updated 7 April). Does Anthropic crawl data from the web, and how can site owners block the crawler? (Help Center). https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
Bajemon, E., & Rochet, A.-L. (2026, 7 September). Scoring Without the Engine: Validating a Deterministic, Manipulation-Resistant Content Score for Generative Engines, End to End. arXiv:2609.07559 (preprint, v1). https://arxiv.org/abs/2609.07559
Brave. (2026). Brave Search Crawler (Brave Search Help, read 9 October 2026). https://search.brave.com/help/brave-search-crawler
Canel, F., & Madhavan, K. (2025, 31 July). Keeping Content Discoverable with Sitemaps in AI Powered Search. Bing Webmaster Blog. https://blogs.bing.com/webmaster/2025/7/Keeping-Content-Discoverable-with-Sitemaps-in-AI-Powered-Search/
Deda, Y. (2025, 24 November). How to Optimize for ChatGPT: Skip LLMs.txt, Earn Trust on Quora and Reddit. SE Ranking. https://seranking.com/blog/chatgpt-citation-factors/
Deda, Y. (2025, 15 December). How to Optimize for AI Mode: Google Visibility Matters 3x More Than Content. SE Ranking. https://seranking.com/blog/how-to-optimize-for-ai-mode/
Fang, H., Tao, S., Chen, N., Chang, K.-X., & Sakai, T. (2025). Do Large Language Models Favor Recent Content? A Study on Recency Bias in LLM-Based Reranking. arXiv:2509.11353 (v2, 21 September 2026). https://arxiv.org/abs/2509.11353
Gavoyannis, D. (2025, 3 December). Short vs. Long Content in AI Overviews: The Data Says Both Work. Ahrefs. https://ahrefs.com/blog/short-vs-long-content-in-ai-overviews/
GoGoChimp. (2026, 28 September). Bing Webmaster Tools AI Performance overview and AI search queries exports for gogochimp.com, data through 25 September 2026. The total since 29 April 2026 combines this pull with the 29 July 2026 export. First-party, unpublished.
GoGoChimp. (2026, October). How citable is the web to AI search? A 2026 benchmark of 244 sites (one-engine re-crawl of 299 sites, 7 October 2026). First-party. https://www.gogochimp.com/research/state-of-ai-citability-2026
Google. (2025, updated 10 December). AI features and your website (Google Search Central). https://developers.google.com/search/docs/appearance/ai-features
Google. (2026, updated 10 July). Google's Guide to Optimizing for Generative AI Features on Google Search (Google Search Central). https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
Google. (2026, updated 14 July). Google's common crawlers (Google Search Central). https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
Google. (2026). Search generative AI control (Search Console Help, available worldwide from 31 August 2026). https://support.google.com/webmasters/answer/16908024
Google. (2026). What's new in Google Analytics (release notes entry of 13 May 2026). https://support.google.com/analytics/answer/9164320
Google. (2026). Default channel group (Analytics Help, read 9 October 2026). https://support.google.com/analytics/answer/9756891
Google Cloud. (2026, updated 7 October). Grounding overview (Gemini Enterprise Agent Platform documentation, formerly Vertex AI). https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/grounding/overview
IndexNow.org. (2026). IndexNow FAQ (read 9 October 2026). https://www.indexnow.org/faq
Indig, K. (2026, 17 February). The Science Of How AI Pays Attention. Search Engine Journal (syndicated from Growth Memo, 16 February 2026). https://www.searchenginejournal.com/the-science-of-how-ai-pays-attention/567597/
Indig, K. (2026, 14 April). Shorter, Focused Content Wins In ChatGPT (data collected by AirOps). Search Engine Journal (syndicated from Growth Memo). https://www.searchenginejournal.com/shorter-focused-content-wins-in-chatgpt/571857/
Kim, S., Jeong, W., Kim, S., Lee, S., & Lee, D. (2026). SAGEO Arena: A Realistic Environment for Evaluating Search-Augmented Generative Engine Optimization. KDD 2026. arXiv:2602.12187 (v2, 7 August 2026). https://arxiv.org/abs/2602.12187
Kirsten, E., Grosse Perdekamp, J., Wu, Q., Upadhyay, M., Gummadi, K. P., & Zafar, M. B. (2026). Characterizing Web Search in The Age of Generative AI. arXiv:2510.11560 (v2, 31 May 2026). https://arxiv.org/abs/2510.11560
Law, R. (2025, 28 July). New Study: AI Assistants Prefer to Cite "Fresher" Content (17 Million Citations Analyzed). Ahrefs. https://ahrefs.com/blog/do-ai-assistants-prefer-to-cite-fresh-content/
Linehan, L. (2025, 11 August). Only 12% of AI Cited URLs Rank in Google's Top 10 for the Original Prompt. Ahrefs. https://ahrefs.com/blog/ai-search-overlap/
Linehan, L. (2026, 15 April). Why ChatGPT Cites One Page Over Another (Study of 1.4M Prompts). Ahrefs. https://ahrefs.com/blog/why-chatgpt-cites-pages/
Linehan, L. (2026, 11 May). We Tracked 1,885 Pages Adding Schema. AI Citations Barely Moved. Ahrefs. https://ahrefs.com/blog/schema-ai-citations/
Madhavan, K. (2025, 8 October). Optimizing Your Content for Inclusion in AI Search Answers. Microsoft Advertising Blog. https://about.ads.microsoft.com/en/blog/post/october-2025/optimizing-your-content-for-inclusion-in-ai-search-answers
Madhavan, K., & Merchant, M. (2026, 23 March). The AI Performance dashboard: Your view into where your brand appears across the AI web. Microsoft Advertising Blog. https://about.ads.microsoft.com/en/blog/post/march-2026/the-ai-performance-dashboard-your-view-into-where-your-brand-appears-across-the-ai-web
Madhavan, K., Merchant, M., Canel, F., & Nigam, S. (2026, 10 February). Introducing AI Performance in Bing Webmaster Tools Public Preview. Bing Webmaster Blog. https://blogs.bing.com/webmaster/2026/2/Introducing-AI-Performance-in-Bing-Webmaster-Tools-Public-Preview/
Madhavan, K., Merchant, M., Nigam, S., & Shah, T. (2026, 16 June). New AI Visibility Insights in Bing Webmaster Tools: Intents, Topics, Citation Share, Compare. Bing Search Blog. https://blogs.bing.com/search/2026/6/New-AI-Visibility-Insights-in-Bing-Webmaster-Tools-Intents-Topics-Citation-Share-Compare/
Madhavan, K., Risvik, K., & Merchant, M. (2026, 6 May). Evolving role of the index: From ranking pages to supporting answers. Bing Search Blog. https://blogs.bing.com/search/2026/5/Evolving-role-of-the-index-From-ranking-pages-to-supporting-answers/
Maoz, H., & Samet, M. (2026, 3 June, with a worldwide note added 31 August). Introducing Search Generative AI performance reports in Search Console. Google Search Central Blog. https://developers.google.com/search/blog/2026/06/gen-ai-performance-reports
Microsoft. (2026). Which crawlers does Bing use? (Bing Webmaster Help, read 9 October 2026). https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0
Microsoft Learn. (2026, 6 March, updated 11 July). How to use Grounding with Bing Search in Foundry Agent Service (classic). https://learn.microsoft.com/en-us/azure/foundry-classic/agents/how-to/tools-classic/bing-grounding
Ong, S. Q. (2025, 20 August). Is ChatGPT Really Powered by Google? 118,931 Fan-Out Queries Analyzed. Ahrefs. https://ahrefs.com/blog/is-chatgpt-really-powered-by-google/
OpenAI. (2026, updated 21 August). Searching the web with ChatGPT (Help Center). https://help.openai.com/en/articles/9237897-chatgpt-search
OpenAI. (2026). Web search (API documentation, read 9 October 2026). https://developers.openai.com/api/docs/guides/tools-web-search
OpenAI. (2026). Overview of OpenAI Crawlers (API documentation, read 9 October 2026). https://developers.openai.com/api/docs/bots
OpenAI. (2026, updated 29 August). Publishers and Developers FAQ (Help Center). https://help.openai.com/en/articles/12627856-publishers-and-developers-faq
Perplexity. (2026). Perplexity Crawlers (documentation, read 9 October 2026). https://docs.perplexity.ai/docs/resources/perplexity-crawlers
Perplexity Research. (2025, 25 September). Architecting and Evaluating an AI-First Search API. https://www.perplexity.ai/hub/blog/architecting-and-evaluating-an-ai-first-search-api
Petrovic, D. (2025, 20 December). How big are Google's grounding chunks? DEJAN AI. https://dejan.ai/blog/how-big-are-googles-grounding-chunks/
Puerto, H., Gubri, M., Green, T., Oh, S. J., & Yun, S. (2025). C-SEO Bench: Does Conversational SEO Work? NeurIPS 2025 Datasets and Benchmarks. arXiv:2506.11097 (v3, 20 October 2025). https://arxiv.org/abs/2506.11097
Raghavan, P. (2020, 15 October). How AI is powering a more helpful Google. Google. https://blog.google/products/search/search-on/
Rizk, I. (2026, 13 May). Citations in Microsoft Clarity Now Generally Available. Microsoft Clarity Blog. https://clarity.microsoft.com/blog/citations-now-generally-available/
Selvam, S., & Ghosh, A. (2026, 14 September). CITECHOICE: A Causal Audit of How Document Presentation Redistributes Citation Credit in Agentic Search. arXiv:2609.15164 (preprint, v1). https://arxiv.org/abs/2609.15164
Taylor, D. (2025, 30 December). Research: Does Structuring Your Content Improve the Chances of AI Mode Surfacing? SALT.agency. https://salt.agency/blog/research-does-structuring-your-content-improve-the-chances-of-ai-mode-surfacing/
Vishwakarma, R., Kumar, S., & Jamidar, R. (Sprinklr). (2026, 25 May). What Gets Cited: Competitive GEO in AI Answer Engines. SIGIR 2026. arXiv:2605.25517. https://arxiv.org/abs/2605.25517
Willison, S. (2025, 21 March). Anthropic Trust Center: Brave Search added as a subprocessor. https://simonwillison.net/2025/Mar/21/anthropic-use-brave/
Zecchini, G., Moore, A. A., Ubl, M., & Siddle, R. (2024, 17 December). The rise of the AI crawler. Vercel, with MERJ. https://vercel.com/blog/the-rise-of-the-ai-crawler

How to Optimise Content for AI Retrieval Pipelines (What 47,000+ Copilot Citations Show)

What is an AI retrieval pipeline?

What do retrieval and reranking do to your page?

Which index does each AI engine retrieve from?