How to Get Content Indexed by AI Search Engines (and Check AI Crawlers Aren't Blocked)

An AI engine can only cite a page its crawler has fetched, and plenty of sites stop the crawler at the door. On 13 October 2026, 11% of the top 1,000 websites blocked OAI-SearchBot, the crawler behind ChatGPT search, in robots.txt (Known Agents). A firewall or CDN can turn it away too, even when robots.txt says yes.

Being cited at all proves the crawler got in. In Bing's sample, Microsoft Copilot cited GoGoChimp 358 times for "how to get content indexed by AI search engines" in the 90 days to 25 September 2026, so Bingbot reaches our pages. That was only 2.53% of the citations Bing gave for the query, so our problem lies in the page and the competition. If an engine never cites you at all, check access before you rewrite a word.

Two switches decide whether a crawler gets in: robots.txt, and the firewall or CDN in front of your site. A robots.txt that says Allow behind a firewall that says no is a chippy with the Open sign up and the shutter down. Then the page has to be indexable, which the rest of our AI SEO guides take for granted.

Why can't AI engines cite a page their crawlers can't reach?

Each engine answers from a search index. A page gets in when that engine's crawler, or a search partner's, fetches it. Block the crawler and the engine can't read your page, though it can still link to you.

Where does each AI engine get its pages?

• ChatGPT search uses OAI-SearchBot and "sometimes partners with other search providers". To be eligible, OpenAI says to allow OAI-SearchBot and "confirm that the website host or content delivery network allows traffic from OpenAI's published searchbot IP addresses" (OpenAI). OpenAI's list of those providers' privacy policies includes Microsoft's, which is one more reason to keep Bing's index current.
• Claude indexes pages with Claude-SearchBot and fetches them live with Claude-User (Anthropic, 2026). For web search, Anthropic lists Brave Search among its subprocessors (Anthropic), so check your pages turn up in Brave too. Our Claude and Gemini SEO guide covers both engines.
• Google AI Overviews and AI Mode need a page that is "indexed and eligible to be shown in Google Search with a snippet" (Google, 2025). Google's July 2026 guide adds that a site "must be included in Search generative AI features in Search Console to be eligible" (Google, 2026).
• Perplexity runs its own index, which it says "tracks over 200 billion unique URLs" (Perplexity, 2025). PerplexityBot fills it. More in our Perplexity SEO guide.
• Copilot doesn't have a bot of its own on Bing's crawler list (Bing). Bing says removed URLs drop out of "Copilot experiences that rely on Bing's index" (Bing). Bing's guidelines say Copilot relies on "the same core crawling, indexing, and ranking foundation as traditional search" (Bing), so Bingbot does Copilot's crawling.
• Apple, Meta, Amazon and DuckDuckGo run their own search crawlers or fetchers: Applebot, Meta-WebIndexer, Amzn-SearchBot and DuckAssistBot. Meta says allowing Meta-WebIndexer "helps us cite and link to your content in Meta AI's responses" (Meta).

Which AI crawlers should you allow, and which can you block?

Allow the search crawler and answer-time fetcher of every engine you want citations from. Search crawlers build the index ahead of time, and answer-time fetchers open a page when a user asks. Training crawlers are a separate call. OpenAI says "Each setting is independent of the others" (OpenAI), so blocking GPTBot leaves ChatGPT search alone.

User agent	Owner	Group	What blocking it does	Vendor doc
OAI-SearchBot	OpenAI	Search crawler	Out of ChatGPT search answers, though your site can still appear as a link	OpenAI
Claude-SearchBot	Anthropic	Search crawler	Not indexed for Claude's search, which "may reduce" your visibility	Anthropic
PerplexityBot	Perplexity	Search crawler	Not indexed to surface and link in Perplexity	Perplexity
Googlebot	Google	Search crawler	Not crawled for Google Search, AI Overviews or AI Mode	Google
Bingbot	Microsoft	Search crawler	Not crawled for Bing or Copilot	Bing
Applebot	Apple	Search crawler	Out of search results in Spotlight, Siri and Safari	Apple
Meta-WebIndexer	Meta	Search crawler	Less likely to be cited and linked in Meta AI	Meta
Amzn-SearchBot	Amazon	Search crawler	Not eligible for search experiences such as Alexa	Amazon
ChatGPT-User	OpenAI	Answer-time fetcher	ChatGPT can't open the page mid-answer. Robots.txt "may not apply"	OpenAI
Claude-User	Anthropic	Answer-time fetcher	Claude can't fetch the page for a user. Honours robots.txt	Anthropic
Perplexity-User	Perplexity	Answer-time fetcher	Perplexity can't fetch the page for a user. "Generally ignores robots.txt rules"	Perplexity
GPTBot	OpenAI	Training crawler	Not used to train OpenAI's models. ChatGPT search unaffected	OpenAI
ClaudeBot	Anthropic	Training crawler	Excluded from Anthropic's training data. Claude-SearchBot and Claude-User unaffected	Anthropic
Google-Extended	Google	Training token	No Gemini training or Gemini app grounding. Google Search and AI Overviews unaffected	Google
Applebot-Extended	Apple	Training token	No Apple model training. Still discoverable in Spotlight, Siri and Safari	Apple
DuckAssistBot	DuckDuckGo	Answer-time fetcher	Out of DuckDuckGo's AI-assisted answers. Organic results unaffected	DuckDuckGo
Meta-ExternalAgent	Meta	Training crawler	Not crawled to train Meta's AI models	Meta
Amazonbot	Amazon	Training crawler	Not crawled to improve Amazon's services or train its AI models	Amazon
CCBot	Common Crawl	Training (our label)	Out of Common Crawl's open web archive	Common Crawl

Common Crawl calls CCBot's output an open archive, so "training" is our label.

Do AI crawlers obey robots.txt?

The search and training crawlers in the table say robots.txt controls them. OpenAI, Perplexity, Google, Meta and Amazon say it may not bind their answer-time fetchers. Anthropic says all three of its bots honour it, and that they "will not attempt to bypass CAPTCHAs", so a CAPTCHA wall keeps Claude out too.

RFC 9309 is the robots.txt standard. Martijn Koster, who first defined the protocol in 1994, wrote it with Google's Gary Illyes, Henner Zeller and Lizzi Sassman. The RFC says the rules "are not a form of access authorization" (IETF, 2022), so if you want a bot kept out, that's the firewall's job.

How many sites block AI search crawlers?

Fewer block the search crawlers than the training ones. Known Agents, which scans the robots.txt files of the top 1,000 websites daily, counted 21% blocking GPTBot and 11% blocking OAI-SearchBot on 13 October 2026 (Known Agents, Known Agents). News sites block far more. Across the top 50 US and top 50 UK news sites in December 2025, BuzzStream found 62% blocking GPTBot and 49% blocking OAI-SearchBot (BuzzStream, 2025).

A Disallow line isn't enforcement, either. HasData found that 234 of 592 sites disallowing GPTBot (39.5%) still served it a page in July 2026 (HasData, 2026). And blocking doesn't always keep a site out of AI answers. In BuzzStream's March 2026 citation data, 82.4% of the news sites that block OAI-SearchBot still appeared in AI citations across the engines it tracked (BuzzStream, 2026).

How do you check AI crawler access with Rubric?

Open the Technical tab in your Rubric report and choose AI crawlers. It reads each AI bot's robots.txt status, sends a live request as six bots to catch firewall blocks robots.txt can't show, and counts noindexed pages. The Crawler reachability card in the Overview shows allowed pages per bot.

Rubric predicts citability, it doesn't count citations.

GoGoChimp's hosted Rubric report of 5 October 2026 found all 15 AI bots it checks allowed in robots.txt, a 200 from all six live probes, and 3 of 182 pages noindexed.

What does a bad result look like?

• A search crawler or answer-time fetcher blocked in the robots.txt matrix.
• A probe returning 403 instead of 200, which fails the "AI crawlers not network-blocked" check.
• A Crawler reachability count below your page total. Ours shows GPTBot and PerplexityBot allowed on 182 of 182 pages.
• Noindexed pages you meant to keep, which the Indexable check lists.

The matrix tags each bot as citation or training. For what each bot does, follow the vendor docs linked in the table above. The live probe doesn't include Claude-SearchBot or Claude-User, so check those two yourself.

Is a 200 from the live probe proof the real bot gets through?

No. Rubric sends each bot's user agent from its own IP address, and a CDN doesn't take a user agent on trust. Cloudflare checks for "a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS" (Cloudflare).

So the probe catches user-agent rules but won't see IP checks. And because Cloudflare says "non-verified bots are still default blocked" (Cloudflare, 2026), a probe can hit a block the real bot never sees. It's a fast first check. Confirm it in your CDN's bot settings and logs.

How do you check crawler access with Claude and the Rubric MCP?

Connect Rubric's hosted MCP to Claude, then pull the crawler results with get_audit, the failing pages with get_affected_pages and any URL with get_page. Reads are free, and a fresh crawl runs a live audit of your site. Rubric's MCP page gives you the Claude Code command with your key filled in.

Use list_audits to find my latest Rubric audit for example.com, then get_audit and report the crawler reachability results. Run get_affected_pages for the "AI crawlers not network-blocked" and Indexable checks, then get_page on each URL. For each page, say whether the cause looks like robots.txt, the firewall, noindex or a canonical tag, and give me the line to change.

How do you fix blocked AI crawlers and unindexed pages?

Work in order: robots.txt, then the firewall or CDN, then indexing. A clean robots.txt won't help if the CDN turns the bot away, and an open firewall won't help if the page carries noindex.

What should robots.txt say for AI crawlers?

To keep AI search and opt out of training, give each training crawler its own group:

# Opt out of AI training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Leave out the Google-Extended group if you want Gemini app answers grounded in your pages, as it controls that too. Meta-ExternalAgent and Amazonbot can join the list if you don't want Meta or Amazon training on your pages.

A User-agent: * group with Disallow: / shuts out every compliant crawler that doesn't have its own group. Give each search crawler and answer-time fetcher you want a group of its own:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

A bot with its own group ignores the * group, since RFC 9309 sends crawlers there "If no matching group exists" (IETF, 2022). So copy across any Disallow lines you'd still want it to follow. Apple's Applebot "will follow Googlebot instructions" when your rules don't mention Applebot (Apple, 2026). Add Applebot, Meta-WebIndexer, Amzn-SearchBot and DuckAssistBot the same way if you want to appear in Apple, Meta AI, Alexa and DuckDuckGo answers.

How often does a firewall stop AI crawlers?

About one site in 20, in our crawls. On 6 October 2026, 25 of 484 agency sites served our crawler a firewall challenge instead of content, and so did 13 of 299 other websites the next day. Another 26 agency sites refused every bot we sent, Googlebot included, which points to a firewall rule rather than an AI policy. Our probes don't come from the bots' own networks, so treat a refusal as a reason to check, not as proof. On Cloudflare's Free and Pro plans, security events now go back at least 31 days, so you can look for blocked AI crawlers over a full month (Cloudflare, 2026).

How do you stop a firewall or CDN blocking AI search bots?

Check your CDN's AI-bot settings, then its logs. Google's advice for AI features includes "Ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure" (Google, 2025).

On Cloudflare, since 15 September 2026, the Block and Block on pages with ads settings "now apply to mixed-use crawlers, including Applebot, Bingbot, and Googlebot". Cloudflare's advice is "To stop training and keep search, use Disallow AI Training" (Cloudflare, 2026). Its recommended preset for new ad-funded domains blocks Agent bots on ad pages (Cloudflare, 2026). Zones that already had Block were moved: "Previous Training selections of Block or Block on pages with ads will migrate to Disallow AI Training" (Cloudflare, 2026), so the risk is in picking Block now. ChatGPT-User counts as an Agent bot (Cloudflare, 2026). Cloudflare's docs page hasn't caught up yet.

Cloudflare can also write AI rules into your robots.txt. Its managed robots.txt reached over 3.8 million domains by its count (Cloudflare, 2025), and Bot Preference Sync now does the job. Cloudflare says "For all new customers, Bot Preference Sync will be on by default", and it adds its rules to your file so "any existing Disallow directives are maintained" (Cloudflare, 2026). Read your live /robots.txt, because it isn't always your CMS copy.

AWS WAF Bot Control's AI category blocks AI bots "regardless of whether the bots are verified or unverified" once switched on (AWS). Vercel's AI-bot ruleset is "inactive by default" (Vercel). Set it to Deny and it "blocks all traffic identified as coming from AI bots", search crawlers included, and "When new AI bots emerge, Vercel automatically adds them to its managed list" with the same action (Vercel). On other CDNs, check each search crawler by name.

I'd use robots.txt for preferences and the firewall for bots you want gone. Anthropic warns that IP blocking "may not work correctly or persistently guarantee an opt-out", because it stops its bots reading robots.txt (Anthropic, 2026).

How do you keep pages indexable for AI search?

Check templates for a stray noindex, and point canonicals at the version you want cited.

• Keep dead URLs out of your sitemap. In Vercel's 2024 analysis, "ChatGPT spends 34.82% of its fetches on 404 pages" (Vercel, 2024).
• Put the main text in the HTML. The same Vercel study found "none of the major AI crawlers currently render JavaScript", and Bing warns that "Content that cannot be reliably rendered may not be indexed or selected for grounding results" (Bing). Our Crawler reachability card shows 27 pages with schema only after JavaScript.
• In Search Console, check the Search generative AI setting is on Include, which Google says is the default (Google). The setting reached every site on 31 August 2026, and excluded content leaves AI Overviews and AI Mode "within 1-2 days". Rubric can't see this setting. Our Google AI Mode SEO guide covers Google's side.
• Look for nosnippet or a tight max-snippet. Google says nosnippet will "prevent the content from being used as a direct input for AI Overviews and AI Mode" (Google). On Bing, NOARCHIVE "prevents content from being used in Copilot responses and grounding results" (Bing).

To take a page out, use noindex and leave it crawlable. Bing says "Make sure Bingbot can access the URL so the directive can be seen" (Bing). Google says that if robots.txt disallows a page, "any information about indexing or serving rules will not be found and will therefore be ignored" (Google). So don't pair a Disallow with noindex on the same URL.

How do you get new pages into Bing and Copilot faster?

Submit your XML sitemap in Bing Webmaster Tools and switch on IndexNow. Bing accepts "Up to 50,000 URLs per individual sitemap file" and revisits sitemaps "typically at least once per day" (Bing, 2025). IndexNow notifies Bing and six other participants, Amazon among them, up to 10,000 URLs per request (IndexNow). Google isn't one of them (IndexNow). Bing says IndexNow "helps ensure that AI systems reference the most current version of a page" (Bing, 2026). Bing's own URL Submission tools "may be deprecated in the future as IndexNow adoption expands" (Bing). A ping isn't a promise, though: "Submitting a URL does not guarantee immediate indexing" (IndexNow).

Google doesn't use IndexNow. Keep your sitemap's lastmod honest, because Google uses it only when it's "consistently and verifiably" accurate (Google), and use Request indexing in Search Console's URL Inspection for pages that can't wait. Google says indexing "typically takes only a day or so, but can take much longer in some cases" (Google).

How do you edit robots.txt on Webflow, Shopify, Squarespace and WordPress?

• Webflow: Site settings, then SEO, then Indexing (Webflow). We're on Webflow, and its "AI bots" toggle groups PerplexityBot, ChatGPT-User and Claude-User with training bots (Webflow). Using it to block training blocks Perplexity's search crawler too, so you're better off writing rules by hand.
• Shopify: change the default file through the robots.txt.liquid template (Shopify), then check the live file for AI-bot rules. Shopify also handles bot management itself, at the network layer, for every store (Shopify), and since May 2026 "Bots and agents that don't sign their requests are subject to the strictest limits" (Shopify, 2026). Its agentic storefronts, which share product data with AI assistants, are "active by default for eligible stores" (Shopify).
• Squarespace: the setting that blocks known AI crawlers is off by default ("We default to having the box unchecked"), but its list includes DuckAssistBot, DuckDuckGo's AI-answer fetcher, as well as training bots (Squarespace, 2026).
• WordPress: ticking "Discourage search engines from indexing this site" adds a sitewide noindex (WordPress.org). That takes the whole site out of Google's AI features and Copilot. On WordPress.com, "Prevent third-party sharing" "adds known AI bots to the 'disallow' list in your site's robots.txt file" (WordPress.com). On one WordPress.com site we checked on 14 October 2026, that list included PerplexityBot, Perplexity's search crawler.

How do you confirm AI crawlers can reach your pages?

Give each change a day, then re-crawl and compare the audits. A re-crawl can't show whether the real bots came back, so check your CDN logs and the engines' answers too.

1. Run start_crawl and crawl_status, then compare_audits for score_delta, fixed and regressed. The /mcp__rubric__fix prompt runs Rubric's side of this loop in Claude.
2. On Cloudflare, AI Crawl Control is "Available on all plans" and lets you "Monitor robots.txt compliance" (Cloudflare).
3. Ask ChatGPT and Perplexity a question your page answers. ChatGPT adds utm_source=chatgpt.com to referral URLs, according to OpenAI's publisher FAQ (archived 4 October 2026), which OpenAI has since redirected, so you'll see those visits in analytics. Our guide to AI visibility tracking and our 2026 AI SEO tools round-up cover doing this at scale.
4. For Copilot, read the AI Performance report in Bing Webmaster Tools, which Bing calls "a sample of overall citation activity" (Bing, 2026). Since 13 October 2026 its Topics Portfolio beta also lists "the strongest queries across your whole portfolio where other sites appear and yours does not" (Bing, 2026).
5. For Google, Search Console's generative AI report counts your AI Overviews and AI Mode impressions. It shows impressions only, with no clicks (Google).
6. If your CDN is Cloudflare, Fastly or CloudFront, Microsoft Clarity's AI Bot Activity report reads your server-side logs (Microsoft Clarity, 2026) and flags "bots accessing disallowed paths in violation of robots.txt" (Microsoft Clarity, 2026).

If your robots.txt is clean and an engine still never cites you, don't rewrite a word until you've checked the firewall.

After that, look at the follow-up questions your pages don't answer, which our long-tail AEO guide covers. To check your own site, run a Rubric audit on a free account.

Questions about AI crawlers and indexing

Can ChatGPT still show your site if you block OAI-SearchBot?

Yes, as a link. OpenAI says opted-out sites "can still appear as navigational links" (OpenAI). OpenAI's publisher FAQ went further before OpenAI redirected it: a disallowed page found through a third-party search provider could still show as a link and page title, and only noindex on a crawlable page stopped that (OpenAI, archived 4 October 2026). Blocking stops ChatGPT reading your page. It doesn't stop it repeating what other sites say about you.

How long does a robots.txt change take to reach AI search?

About a day, where engines publish a figure. OpenAI says about 24 hours for ChatGPT search (OpenAI), and Perplexity says up to 24 hours (Perplexity). Google caches robots.txt "for up to 24 hours" (Google). Page-level changes can take longer on Google, which says recrawling a page "can take anywhere from several days to several months" (Google, 2025).

Do you need llms.txt to get indexed by AI search engines?

Not for Google, which says you don't need "new machine readable files, AI text files, markup, or Markdown" to appear in its search, generative AI features included (Google, 2026). The file isn't an access control either, so it can't unblock a crawler. Our llms.txt explainer covers what it does do.

How do you get JavaScript or code content indexed by AI search engines?

Put it in the HTML as text. Bing says it renders with "the most recent stable version of Microsoft Edge" (Bing). Google processes JavaScript "as long as it isn't blocked" (Google, 2026). Vercel's 2024 data found no rendering by OpenAI's, Anthropic's or Perplexity's crawlers. A January 2026 test put the practical rendering rate of OpenAI's crawlers at 0% (EdgeComet, 2026). One site owner reported OpenAI's crawlers rendering some pages from late September 2026, but OpenAI hasn't said so. Keep code samples as text in pre blocks, not screenshots.

What happens if robots.txt returns an error?

A server error locks compliant crawlers out, and a missing file lets them in. RFC 9309 says a 5xx means "the crawler MUST assume complete disallow", while a 4xx means it "MAY access any resources on the server" (IETF, 2022). Google says that for the first 12 hours it "stops crawling the site but keeps trying to fetch the robots.txt file" (Google). So check that your firewall and host don't answer /robots.txt with a 5xx.

Can you submit your site to ChatGPT, Claude or Perplexity?

No. None of them offers a submission form or a console. Their docs say to allow their crawlers, and Google's John Mueller says AI crawlers "usually don't have a console or any setup where you can submit a sitemap file", so keep your sitemap at the standard sitemap.xml address (Search Off the Record, 2026). For Copilot, use IndexNow and Bing Webmaster Tools.

Can AI search engines index PDFs?

Google can. It converts a PDF to HTML during indexing, "right after rendering", and says "indexing and ranking is pretty much the same as any other document" (Search Off the Record, 2026). It refetches PDFs far less often than HTML, so when you change one, Google says "it can make sense to just use a new file name". A scanned PDF hides its text, so check you can select and copy the words. OpenAI, Anthropic and Perplexity don't document how they handle PDFs, so publish anything you want cited as an HTML page too.

Does noindex keep a page out of Perplexity and Claude?

We couldn't find anything on noindex in Perplexity's or Anthropic's crawler docs, though Google and Bing document it. To keep a page out of those two, disallow PerplexityBot, Claude-SearchBot and Claude-User on that path, which leaves Googlebot free to read your noindex. Perplexity-User "generally ignores" robots.txt, so put private pages behind a login.

References

• Allen, W. (2025, 24 September). Cloudflare blog post on the Content Signals Policy and managed robots.txt. https://blog.cloudflare.com/content-signals-policy/
• Amazon Web Services. (n.d.). AWS WAF Bot Control rule group (AWS WAF Developer Guide). https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-bot.html
• Amazon. (n.d.). Amazonbot (Amazon Developer). https://developer.amazon.com/amazonbot
• Anthropic. (2026, 7 April). Does Anthropic crawl data from the web, and how can site owners block the crawler? (Anthropic Help Center). https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
• Anthropic. (n.d.). Subprocessors (Anthropic Trust Center). https://trust.anthropic.com/subprocessors
• Apple. (2026, 4 September). About Applebot (Apple Support). https://support.apple.com/en-us/119829
• Becker, B. (2026, 15 September). Cloudflare blog post on accountable mixed-use AI crawlers and the new AI-bot settings. https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/
• Becker, B., & Lee, J.-H. (2026, 1 July). Cloudflare blog post on Search, Agent and Training options for AI crawlers. https://blog.cloudflare.com/content-independence-day-ai-options/
• Bing. (n.d.). Bing Webmaster Guidelines (Bing Webmaster Tools Help). https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a
• Bing. (n.d.). How do I Permanently Remove a URL or Page from Bing or Copilot? (Bing Webmaster Tools Help). https://www.bing.com/webmasters/help/how-to-permanently-remove-a-url-or-page-from-bing-or-copilot-37c07477
• Bing. (n.d.). URL Submission (Bing Webmaster Tools Help). https://www.bing.com/webmasters/help/url-submission-62f2860b
• Bing. (n.d.). Which Crawlers Does Bing Use? (Bing Webmaster Tools Help). https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0
• BuzzStream. (2025, 18 December). Which AI bots do top news sites block? https://www.buzzstream.com/blog/publishers-block-ai-study/
• BuzzStream. (2026, 18 March). Do news sites that block AI bots still get cited? https://www.buzzstream.com/blog/news-block-ai-bots-citations/
• Canel, F., & Madhavan, K. (2025, 31 July). Keeping Content Discoverable with Sitemaps in AI-Powered Search. Bing Webmaster Blog. https://blogs.bing.com/webmaster/2025/7/Keeping-Content-Discoverable-with-Sitemaps-in-AI-Powered-Search/
• Cloudflare. (2026, 2 October). 30 days of analytics data on every plan (Cloudflare changelog). https://developers.cloudflare.com/changelog/post/2026-10-02-30-days-analytics-on-every-plan/
• Cloudflare. (2026, 21 August). Bot Preference Sync (Cloudflare blog). https://blog.cloudflare.com/bot-preference-sync/
• Cloudflare. (2026, updated 14 August). AI Crawl Control (Cloudflare docs). https://developers.cloudflare.com/ai-crawl-control/
• Cloudflare. (n.d.). Verified bots (Cloudflare docs). https://developers.cloudflare.com/bots/concepts/bot/verified-bots/
• Cloudflare. (n.d., read 5 October 2026). Block AI bots (Cloudflare docs, still showing its pre-15 September wording). https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/
• Common Crawl. (n.d.). CCBot. https://commoncrawl.org/ccbot
• DuckDuckGo. (n.d.). DuckAssistBot (DuckDuckGo Help). https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot
• EdgeComet. (2026, 15 January). OpenSEOTest: how GPTBot and ChatGPT-User handle JavaScript. https://edgecomet.com/blog/openseotest-how-gptbot-and-chatgpt-user-handle-javascript/
• GoGoChimp. (2026, 28 September). Bing Webmaster Tools AI Performance overview and AI search queries exports for gogochimp.com, data through 25 September 2026. First-party, unpublished.
• GoGoChimp. (2026, 5 October). Rubric report for gogochimp.com, hosted crawl (182 pages): AI crawlers tab, Crawler reachability card and Indexable check. First-party.
• Google Search Central. (2026, 1 October). Search Off the Record, episode 114 (transcript). https://traffic.libsyn.com/secure/search-off-the-record/Search_Off_the_Record_episode_114_transcript.pdf
• Google Search Central. (2026, 13 October). Search Off the Record, episode 115: PDFs in Search (transcript). https://traffic.libsyn.com/secure/search-off-the-record/Search_Off_the_Record_episode_115_transcript.pdf
• Google. (2025, updated 10 December). AI features and your website (Google Search Central). https://developers.google.com/search/docs/appearance/ai-features
• Google. (2026, updated 10 July). Optimizing your website for generative AI features on Google Search (Google Search Central). https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
• Google. (2026, updated 14 July). Google's common crawlers. https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
• Google. (2026, updated 19 August). List of Google user-triggered fetchers. https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers
• Google. (2026, updated 31 August). How Google interprets the robots.txt specification. https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec
• Google. (n.d.). Ask Google to recrawl your URLs / URL Inspection tool (Search Console Help). https://support.google.com/webmasters/answer/9012289
• Google. (n.d.). Build and submit a sitemap (Google Search Central). https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap
• Google. (n.d.). Generative AI performance report (Search Console Help). https://support.google.com/webmasters/answer/16984139
• Google. (n.d.). Robots meta tag, data-nosnippet, and X-Robots-Tag specifications (Google Search Central). https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag
• Google. (n.d., rollout note 31 August 2026). Search generative AI control (Search Console Help). https://support.google.com/webmasters/answer/16908024?hl=en
• HasData. (2026). AI crawler block index (July 2026 baseline, 16 September re-run). https://hasdata.com/blog/ai-crawler-block-index
• IndexNow.org. (n.d.). FAQ. https://www.indexnow.org/faq
• IndexNow.org. (n.d.). searchengines.json (participating search engines). https://www.indexnow.org/searchengines.json
• Known Agents. (2026, read 13 October). GPTBot and OAI-SearchBot agent pages (daily robots.txt scans of the top 1,000 websites). https://knownagents.com/agents/oai-searchbot
• Koster, M., Illyes, G., Zeller, H., & Sassman, L. (2022, September). RFC 9309: Robots Exclusion Protocol. IETF. https://www.rfc-editor.org/rfc/rfc9309.html
• Madhavan, K., Merchant, M., Canel, F., & Nigam, S. (2026, 10 February). Introducing AI Performance in Bing Webmaster Tools Public Preview. Bing Webmaster Blog. https://blogs.bing.com/webmaster/February-2026/Introducing-AI-Performance-in-Bing-Webmaster-Tools-Public-Preview
• Madhavan, K., Nigam, S., & Jain, V. (2026, 13 October). Introducing Topics Portfolio (beta) in AI Performance. Bing Webmaster Blog. https://blogs.bing.com/webmaster/2026/10/introducing-topics-portfolio-beta-in-ai-performance/
• Meta. (2026, updated 21 May). Meta Web Crawlers (Meta for Developers). https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
• Microsoft Clarity. (2026, 22 January). AI Bot Activity in Clarity. https://clarity.microsoft.com/blog/ai-bot-activity-in-clarity/
• Microsoft Clarity. (2026, 26 May). Updates to Bot Activity in Clarity. https://clarity.microsoft.com/blog/updates-to-bot-activity-in-clarity/
• OpenAI. (n.d.). Overview of OpenAI Crawlers. https://developers.openai.com/api/docs/bots
• OpenAI. (n.d.). Searching the web with ChatGPT (OpenAI Help Center). https://help.openai.com/en/articles/9237897-chatgpt-search
• OpenAI. (n.d., archived 4 October 2026). Publishers and Developers - FAQ (OpenAI Help Center, since redirected to its ChatGPT Atlas retirement notice). https://web.archive.org/web/20261004224023/https://help.openai.com/en/articles/12627856-publishers-and-developers-faq
• Perplexity Research. (2025, 25 September). Perplexity blog post on architecting and evaluating an AI-first search API. https://www.perplexity.ai/hub/blog/architecting-and-evaluating-an-ai-first-search-api
• Perplexity. (n.d.). Perplexity Crawlers (Perplexity docs). https://docs.perplexity.ai/docs/resources/perplexity-crawlers
• Shopify. (2026, 7 May). Bots and agents should identify themselves via Web Bot Auth (Shopify developer changelog). https://shopify.dev/changelog/bots-and-agents-should-identify-themselves-via-web-bot-auth
• Shopify. (n.d.). Agentic storefronts (Shopify Help Center). https://help.shopify.com/en/manual/online-sales-channels/agentic-storefronts
• Shopify. (n.d.). Customize robots.txt (Shopify.dev). https://shopify.dev/docs/storefronts/themes/seo/robots-txt
• Shopify. (n.d.). Editing robots.txt (Shopify Help Center). https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt
• Squarespace. (2026, 8 January). Controlling how crawlers access your site (Squarespace Help). https://support.squarespace.com/hc/en-us/articles/360022347072
• Vercel. (2026, updated 10 September). Bot management (Vercel docs). https://vercel.com/docs/bot-management
• Vercel. (2026, updated 10 September). WAF Managed Rulesets (Vercel docs). https://vercel.com/docs/vercel-firewall/vercel-waf/managed-rulesets
• Webflow. (2026, modified 29 April). Disable search engine indexing (Webflow Help). https://help.webflow.com/hc/en-us/articles/33961368603539-Disable-search-engine-indexing
• Webflow. (n.d., read 5 October 2026). Set robots.txt rules (Webflow Help). https://help.webflow.com/hc/en-us/articles/41954080897683-Set-robots-txt-rules
• WordPress.com. (2026, reviewed 5 October). Make your website public (WordPress.com Support). https://wordpress.com/support/privacy-settings/make-your-website-public/
• WordPress.org. (2024, modified 8 June). Settings Reading screen (WordPress documentation). https://wordpress.org/documentation/article/settings-reading-screen/
• Zecchini, G., Moore, A. A., Ubl, M., & Siddle, R. (2024, 17 December). The rise of the AI crawler. Vercel, with MERJ. https://vercel.com/blog/the-rise-of-the-ai-crawler