Cloudflare Configuration for Crawlability, Ads Delivery, and AI Bot Access

Cloudflare can protect a website without making it invisible to search engines, advertising crawlers, product-feed checks, or selected AI search bots. The real SEO risk usually comes from over-broad security rules, challenge pages, cache mistakes, redirect loops, and user-agent-only bot policies. This article explains how to configure Cloudflare so public content remains crawlable while checkout, account, API, and other sensitive paths stay protected.

Practical default: keep Cloudflare in front of public web traffic, use Full (strict) SSL/TLS, and create narrow verified-bot exceptions for public GET and HEAD requests. Do not broadly bypass protection for all bot traffic, and do not put browser-only challenge flows in front of pages that must be crawled by Google Search, Google Ads, Merchant Center, or AI search systems.

Executive summary

Cloudflare does not inherently block search engines, ad systems, or AI bots. The problems usually appear when Cloudflare is configured as if every non-browser request is abuse. Common causes include WAF rules that challenge known crawlers, rate limits that punish legitimate crawling, user-agent-only allowlists that are easy to spoof, cache rules that change cookie or header behavior, and redirect or SSL/TLS setups that create loops.

The safest operating model is simple: public discovery pages should be easy to fetch; private and transactional paths should be hard to abuse. Category pages, product pages, article pages, image assets, JavaScript and CSS required for rendering, JSON-LD, PDFs, robots.txt, and sitemaps should be reachable without sign-in or JavaScript challenge pages. Login, cart, checkout, account areas, mutation APIs, and private endpoints should stay protected.

For Google Search, Google Ads, and Merchant Center, two checks matter most. First, make sure the crawler is allowed in robots.txt and is not blocked by Cloudflare or origin-side bot protection. Second, make sure the landing page returns a stable 200 OK response, does not require login, and does not sit behind a long or unstable redirect chain. Google Ads explicitly expects destination pages to be reachable, and Google notes that landing-page problems can come from firewalls, including Cloudflare.

For AI bots, separate policy from implementation. A site may want visibility in AI search products without allowing training crawlers. OpenAI, Anthropic, and Perplexity publish different bot identities for different purposes. For example, allowing OAI-SearchBot does not require allowing GPTBot. Allowing Claude-SearchBot does not require allowing ClaudeBot. This difference is commercially important for publishers, ecommerce teams, SaaS companies, and brands that want discovery but not unrestricted data reuse.

The practical recommendation is to use Cloudflare verified-bot categories, provider-published IP ranges, reverse DNS verification where appropriate, and narrow WAF Skip rules. Avoid broad IP Access Allow rules unless you fully understand what they bypass. Keep browser-only challenges away from crawlable content. Preserve canonical tags, structured data, X-Robots-Tag, hreflang, and cache semantics. Monitor Security Events, Security Analytics, Bot Analytics, Search Console, Merchant Center, and ad-platform diagnostics after every policy change.

Where Cloudflare typically breaks legitimate crawling

The first failure point is the security stack itself. Cloudflare custom rules, rate limiting, Managed Rules, Super Bot Fight Mode, IP Access Rules, Bot Management, and challenge features can all act before a crawler reaches the origin. A rule that looks harmless for humans may still block or challenge a crawler because the crawler does not behave like a normal browser session.

The second failure point is over-broad allowlisting. IP Access Rules are tempting because they feel simple, but they can bypass more security layers than intended. For crawl safety, Cloudflare custom rules and Skip actions are usually a better first choice because they can bypass only the specific products causing false positives.

The third failure point is challenge-based anti-bot logic. Interstitial challenges and JavaScript detections are designed for browser-like clients. Cloudflare JavaScript Detections work by injecting JavaScript into HTML responses, storing the result in a cf_clearance cookie, and allowing rules to act on fields such as cf.bot_management.js_detection.passed. That can be useful for browser traffic, but it is a poor dependency for non-browser crawlers. A page that must be crawlable should not require JavaScript challenge completion.

The fourth failure point is confusion between crawling and indexing. robots.txt controls crawling, not indexing. If a URL must be removed from search results, use a crawlable noindex directive or another removal method. If a page is blocked in robots.txt, Google may not be able to fetch the page and read its noindex or X-Robots-Tag instruction.

The fifth failure point is cache and session behavior. Cloudflare caching can be very effective for public pages, but account pages, personalized pages, cart flows, checkout flows, and session-sensitive endpoints should not be treated like static content. Headers such as Set-Cookie, cache rules such as “cache everything,” and HEAD-to-GET conversion can all create surprises if the origin or application is fragile.

The sixth failure point is redirect and SSL/TLS inconsistency. Full (strict) is the clean long-term SSL/TLS posture for most production sites. Problems often appear when Cloudflare, origin redirects, HSTS, Workers, Redirect Rules, and Page Rules all try to control the same HTTP-to-HTTPS or hostname canonicalization logic. Search crawlers and ad crawlers fail on the same redirect loops that real users do.

The configuration model that preserves indexing and ad-system access

A crawl-safe Cloudflare configuration starts with a boundary: public acquisition content is not the same as private transaction infrastructure. Public pages should be fetchable without challenge pages. Private paths should remain protected and should usually not be exposed to crawlers at all.

The cleanest Cloudflare pattern is to keep public hostnames proxied through Cloudflare and create narrow exceptions using verified-bot logic, provider IP lists, or both. Cloudflare exposes cf.verified_bot_category for custom rules and rate limiting. Relevant categories include Search Engine, Advertising & Marketing, AI Search, AI Assistant, and AI Crawler. This is more maintainable than a long user-agent list because it targets bot purpose instead of fragile text strings.

For search and ad crawlers, the default posture should be “allow or skip security products on public GET/HEAD pages,” not “let bots do anything everywhere.” Use Skip rules to bypass rate limiting, Managed Rules, or Super Bot Fight Mode only where legitimate crawlers are being caught. Continue to block or challenge unknown automation, fake bots, and requests to sensitive paths.

For AI bots, decide by purpose. Allow AI search bots if you want discovery in AI-generated answers and search experiences. Block training crawlers if you do not want that use case. Treat user-triggered retrieval separately from autonomous crawling. In practice, this means your policy for OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, and PerplexityBot may legitimately differ.

Cloudflare’s managed AI robots.txt feature can help publish AI-crawler preferences, but it is not the same as enforcement. Robots directives are voluntary. If you need enforcement, use WAF logic, AI Crawl Control where available, and provider verification signals.

Cloudflare should reinforce standard technical SEO signals, not replace them. Keep one canonical URL per page, keep sitemaps current, keep hreflang reciprocal, keep structured data in the server-returned HTML where Merchant Center or product freshness matters, and avoid Workers or transforms that cause crawlers to see materially different content from users.

Recommended posture by crawler class

Area	Search crawlers	Ad crawlers	AI bots
Cloudflare security action	Use Skip or narrowly tuned allow logic for verified search bots on public GET/HEAD content. Do not broadly bypass all security everywhere.	Use Skip for verified ad crawlers on landing pages, assets, and product pages. Keep checkout and account paths protected.	Split by purpose: allow AI Search and selected user-triggered assistants where commercially useful; block AI Crawler where training use is not desired.
Challenge pages	Avoid on crawlable pages. Browser-oriented challenges can interrupt crawler access.	Avoid on ad destination URLs and Shopping product pages. Landing pages should not require a challenge or sign-in.	Avoid for AI search bots. Do not rely on JavaScript challenge completion for bot access.
JavaScript Detections	Do not enforce on pages intended for non-browser crawlers.	Do not make ad-quality crawlers complete JavaScript clearance before fetching HTML.	Same rule. Most AI fetchers should be treated as non-browser agents.
robots.txt strategy	Use explicit groups where useful for Googlebot, Googlebot-Image, and other search crawlers. Keep robots.txt on each relevant host.	Add explicit AdsBot-Google and Storebot-Google groups. Do not assume the global * group covers every Google Ads crawler.	Use explicit groups for OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, and PerplexityBot.
Cache and cookies	Cache anonymous public pages; bypass cache on login/session cookies.	Same, especially for price, availability, product images, and landing-page content.	Same for public content. Avoid cache behavior that strips needed headers or changes page meaning.
Canonicals and redirects	Keep one-hop canonical redirects and stable HTTPS.	Keep destination chains short. Google Ads recommends fewer than ten redirects.	Keep fetch targets stable and canonical to reduce duplicate crawling and inconsistent references.
Structured data	Preserve render-critical CSS/JS, canonical tags, robots directives, and structured data.	For Merchant Center, structured data should be in server-returned HTML and match visible values.	Clean HTML, stable semantics, and accessible structure improve machine readability, even when AI bot behavior differs by provider.

Bot-by-bot identification and recommended actions

The practical rule is consistent across providers: never trust user agent alone. Combine user-agent checks with provider-published IP ranges, reverse DNS verification, or Cloudflare’s verified-bot classification where available. Requests that claim to be a known crawler but fail verification should be treated as fake bots.

Bot	Primary use	Known identification methods	Recommended action
Googlebot	Google Search indexing across Search, Images, Video, News, and Discover.	User-agent token plus Google reverse DNS / forward DNS verification or Google-published common crawler IP ranges.	Allow on public crawlable content. Exempt from challenge pages and over-tight rate limits. Do not block render resources needed to understand the page.
AdsBot-Google	Google Ads landing-page and destination quality checks.	User-agent token plus Google special-case crawler verification or published special-crawler IP data. Use an explicit robots group.	Allow on all ad destinations and required assets. Do not require sign-in. Avoid challenge pages. Keep redirect chains short.
Storebot-Google	Google Shopping and Merchant Center product, inventory, and local availability validation.	User-agent token plus Google crawler verification. Do not assume Storebot always uses the same IPs as Google Search crawlers.	Allow on product pages, inventory pages, product images, and relevant purchase or pickup flow surfaces that Merchant Center checks.
OAI-SearchBot	Discovery for ChatGPT search experiences.	User-agent token plus OpenAI-published IP JSON or Cloudflare AI bot detection where available.	Allow if you want visibility in ChatGPT search. Block separately if your policy is not to appear in that channel.
ChatGPT-User	User-triggered fetches from ChatGPT, Custom GPTs, and GPT Actions.	User-agent token plus OpenAI-published IP JSON. OpenAI notes that robots rules may not apply in the same way because requests are user-initiated.	Usually allow for public pages if you want ChatGPT users to access your content. Protect sensitive actions through authentication and app authorization, not through robots assumptions.
GPTBot	OpenAI training and model-improvement crawling.	User-agent token plus OpenAI-published IP JSON or Cloudflare detection where available.	Block unless you explicitly want to allow this use case. Allowing OAI-SearchBot does not require allowing GPTBot.
PerplexityBot	Perplexity search discovery and answer sourcing.	User-agent token plus Perplexity’s official IP list. Perplexity recommends combining user-agent and IP checks in WAF rules.	Allow if you want appearance in Perplexity answers or search surfaces. Verify with both user-agent and official IP data.
ClaudeBot	Anthropic model development and training crawling.	User-agent token plus Anthropic public bots JSON. Anthropic says robots controls are the preferred opt-out method.	Block unless you explicitly want this training use case. If allowing, throttle politely rather than challenge.
Claude-SearchBot	Claude search quality and indexing for search responses.	User-agent token plus Anthropic bots JSON or Cloudflare detection where available.	Allow if you want Claude search visibility; otherwise block explicitly.
Claude-User	User-initiated retrieval on behalf of Claude users.	User-agent token plus Anthropic bots JSON.	Allow if you want user-directed Claude access to public pages. Block if your policy excludes this access.

These differences matter. Google separates common crawlers and special-case crawlers. OpenAI separates search, training, user-triggered browsing, and ad validation. Anthropic separates training, search, and user-triggered retrieval. Perplexity publishes a specific crawler and recommends combined verification. A single “block AI bots” switch is often too crude for marketing teams that want reach without unrestricted training use.

Practical Cloudflare configurations that work

WAF and firewall rules

The modern Cloudflare pattern is to prefer custom rules and Skip actions over legacy user-agent blocking or broad IP Access Allow rules. The reason is control. A Skip rule can bypass only selected products, while a broad IP access allowlist may bypass more protection than you intended.

A solid first rule for SEO and ads is a public-content skip rule. The goal is not to let bots do anything; it is to stop false positives on content that should be crawlable anyway.

Rule name: Skip security for verified search and ad crawlers on public content

Expression:
(
  cf.verified_bot_category in {"Search Engine" "Advertising & Marketing"}
  and http.request.method in {"GET" "HEAD"}
  and not starts_with(http.request.uri.path, "/checkout")
  and not starts_with(http.request.uri.path, "/cart")
  and not starts_with(http.request.uri.path, "/account")
  and not starts_with(http.request.uri.path, "/api/private")
)

Action:
Skip

Skip phases:
- http_ratelimit
- http_request_firewall_managed
- http_request_sbfm

This type of rule is most appropriate on proxied public hostnames where search and ad crawlers must fetch HTML and assets without tripping abuse controls. Keep transactional and private paths outside the exception.

For AI bots, the configuration is usually more selective because policy differs by purpose. A publisher might allow AI Search and block AI Crawler. A SaaS company might allow user-triggered assistant fetches for public docs but block all AI access to application paths. Higher-end Cloudflare Bot Management setups can use detection IDs and richer bot fields; lower-plan setups may rely more heavily on AI Crawl Control and explicit robots.txt groups.

User-agent plus IP rule example

When provider-specific logic is required, combine a user-agent token with an imported IP list. This pattern is useful for OpenAI, Perplexity, Anthropic, and Google special-case crawlers. It is much safer than user-agent matching alone.

Rule name: Allow verified AI search bots on public GET/HEAD

Expression:
(
  http.request.method in {"GET" "HEAD"}
  and not starts_with(http.request.uri.path, "/checkout")
  and not starts_with(http.request.uri.path, "/account")
  and (
    (http.user_agent contains "OAI-SearchBot" and ip.src in $openai_searchbot_ips) or
    (http.user_agent contains "PerplexityBot" and ip.src in $perplexitybot_ips) or
    (http.user_agent contains "Claude-SearchBot" and ip.src in $anthropic_bots_ips)
  )
)

Action:
Skip

This example is intentionally conservative. It only affects public read requests. It leaves checkout, account, and other sensitive paths under the normal security policy. The maintenance burden is in the IP lists, so provider-published JSON endpoints should be imported automatically rather than copied by hand into long-lived rules.

Rate limiting

Rate limiting should protect mutation and abuse surfaces, not punish normal crawl behavior. Exclude verified search and ad bots from content-path rate limits, while keeping aggressive limits on login, password reset, search APIs, quote forms, cart AJAX, and feed-generation endpoints that can be abused.

Do not rate-limit the following patterns without crawler exceptions:

• category and product HTML pages;
• image assets required for Merchant Center;
• CSS and JavaScript required for rendering;
• sitemap files and robots.txt;
• article pages and pagination that search bots normally crawl.

Page Rules, Workers, caching, and session handling

Page Rules still exist, but Cloudflare’s newer Rules products are generally more configurable. Treat Page Rules as a legacy fit for simple redirects or simple cache tuning, not as the center of crawler policy.

Workers are powerful because they can modify requests and responses at the edge. That also makes them risky if they rewrite URLs, headers, cookies, or session behavior differently by user-agent. For crawl safety, use Workers sparingly and predictably: normalize headers, handle CORS or preflight edge cases, or add an X-Robots-Tag to non-HTML assets such as PDFs when origin control is limited.

export default {
  async fetch(request) {
    const response = await fetch(request);
    const url = new URL(request.url);

    // Clone response so headers can be modified safely.
    const out = new Response(response.body, response);

    // Example: ensure crawlable PDFs expose explicit robots instructions.
    if (url.pathname.endsWith(".pdf")) {
      out.headers.set("X-Robots-Tag", "index, follow");
    }

    return out;
  }
};

For caching, the safest marketing-site rule is to cache anonymous public pages and bypass cache on session cookies. This protects crawlable pages without accidentally sharing account-specific or cart-specific content.

Rule name: Bypass cache for authenticated traffic

Expression:
(http.cookie contains "sessionid" or http.cookie contains "wordpress_logged_in" or http.cookie contains "logged_in")

Then:
Cache eligibility = Bypass cache

Keep robots.txt, sitemaps, CSS, JavaScript, images, canonical HTML, and public product or article pages cache-friendly. Keep login, account, cart, checkout, personalized pricing, and geolocated inventory logic out of shared cache paths.

SSL/TLS, redirects, DNS, headers, and CORS

Use Full (strict) unless you have a temporary migration constraint. If you enable “Always Use HTTPS,” make sure the origin is not redirecting HTTPS back to HTTP and that you do not have competing redirect logic in Workers, Redirect Rules, Page Rules, and origin configuration.

For DNS, proxy public web hostnames and keep non-web verification records DNS-only when the third-party service expects direct DNS targets. This matters for domain verification, platform onboarding, certificate workflows, and adjacent marketing infrastructure.

For CORS, remember that Cloudflare can cache and pass along Access-Control-Allow-Origin. If you change origin CORS behavior, you may need a purge or URL change before browsers, crawlers, and QA tools see the updated headers consistently. Public APIs needed for rendering or schema hydration should return predictable GET, HEAD, and OPTIONS behavior.

Robots, canonicalization, sitemaps, structured data, hreflang, and APIs

A crawler-safe robots.txt should be explicit where product-specific behavior exists. Google Ads crawlers may need their own groups. Anthropic supports Crawl-delay; Google does not. OpenAI and Perplexity use different robots tokens for different products. Google also treats robots.txt at the host, protocol, and port level, so every important hostname needs its own accessible file.

Here is a practical robots.txt pattern for a commerce site that wants Google Search, Google Ads, Merchant Center, AI search visibility, and no training crawls:

User-agent: *
Disallow: /checkout/
Disallow: /cart/
Disallow: /account/
Disallow: /search?
Allow: /wp-content/uploads/
Allow: /static/
Sitemap: https://www.example.com/sitemap.xml

User-agent: Googlebot
Allow: /
Disallow: /checkout/
Disallow: /cart/
Disallow: /account/

User-agent: Googlebot-Image
Allow: /media/
Allow: /images/

User-agent: AdsBot-Google
Allow: /
Disallow: /account/
Disallow: /checkout/

User-agent: Storebot-Google
Allow: /
Disallow: /account/
Disallow: /checkout/

User-agent: OAI-SearchBot
Allow: /
Disallow: /account/
Disallow: /checkout/

User-agent: ChatGPT-User
Allow: /
Disallow: /account/
Disallow: /checkout/

User-agent: GPTBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /
Disallow: /account/
Disallow: /checkout/

User-agent: Claude-User
Allow: /
Disallow: /account/
Disallow: /checkout/

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Allow: /
Disallow: /account/
Disallow: /checkout/

This pattern separates public discovery from private workflow and training use. The important caveat is indexing: if a URL must disappear from Google Search, do not rely on robots.txt alone. Use noindex or X-Robots-Tag on a crawlable response until removal is processed.

Canonicalization should stay simple. Choose one canonical URL for each page, redirect alternatives to it, and submit canonical URLs in sitemaps. With Cloudflare in front, that means one HTTPS protocol, one hostname strategy, no mixed www and non-www ambiguity, and no Worker rewrites that leave multiple externally fetchable equivalents alive.

Sitemaps remain one of the cheapest crawl-management tools. Many sites do not need complex crawl-budget throttling; they need clean sitemaps, canonical URLs, fast responses, and no accidental security blocks.

For structured data, Merchant Center is strict in ways many site owners underestimate. Product markup should be present in the HTML returned by the server, should match visible values, and should not vary unpredictably by IP, browser type, cookie state, or geolocation experiment. Do not let Cloudflare caching, edge variation, or cookie-dependent rendering make crawlers see a different price or availability state from buyers.

For hreflang, keep every locale variant directly crawlable, canonicalized to itself, and reciprocally linked. Cloudflare should not geo-redirect crawlers away from alternate language URLs. If you need geo-personalization, preserve stable URLs for search engines and ad QA tools.

For APIs and JSON endpoints, think operationally. JavaScript Detections should not be enforced on API traffic. Public APIs needed for rendering should return predictable responses and correct CORS headers. Private APIs should remain protected and generally excluded from search, ad, and AI bots. If Cloudflare is proxying the hostname, remember that the origin may see Cloudflare as the source IP unless it reads CF-Connecting-IP or X-Forwarded-For.

Monitoring, rollout, and troubleshooting

Cloudflare gives you three useful observability layers for this work. Security Events shows requests actioned or flagged by Cloudflare security products and is the right place to investigate false positives. Security Analytics shows broader request patterns and can help identify rate-limit candidates. Bot Analytics shows bot scores, bot sources, bot tags, and bot decisions. For deeper forensic work, Logpush can deliver HTTP request logs, but it should be enabled before you need historical detail.

A safe rollout should happen in stages rather than as one large firewall change.

Troubleshooting checklist

• [ ] Check the exact HTTP status and whether the request was blocked, challenged, skipped, or rate-limited in Cloudflare Security Events.
• [ ] Confirm that the public URL is reachable without sign-in and returns 200 OK.
• [ ] Verify the crawler as genuine. For Google, use reverse DNS / forward DNS or Google’s published IP ranges. For OpenAI, Anthropic, and Perplexity, use provider-published JSON IP sources. For Cloudflare Bot Management, confirm verified-bot classification or the expected AI bot detection.
• [ ] Inspect robots.txt on the exact host and protocol the bot fetches.
• [ ] Check whether a special crawler needs its own robots group. AdsBot-Google and Storebot-Google should not be treated as an afterthought.
• [ ] Look for X-Robots-Tag or crawler-specific meta robots directives that accidentally de-index the page or asset.
• [ ] Count redirects. Long chains and loops affect users, search crawlers, ad crawlers, and product-feed checks.
• [ ] Review challenge logic. Remove interstitial challenges, Under Attack behavior, and JavaScript Detections enforcement from crawlable pages.
• [ ] Review cache and cookie logic. Bypass shared cache on authenticated cookies and confirm Set-Cookie behavior is not changing crawl semantics.
• [ ] For Merchant Center pages, confirm structured data is present in initial HTML, matches visible values, and does not vary by IP or browser type.

Open questions and limitations

This review prioritizes official Cloudflare, Google, OpenAI, Anthropic, and Perplexity documentation because those are the highest-confidence sources for production configuration decisions. IP ranges for non-Google AI bots are published by the operators and may change. In production, those lists should be imported automatically rather than copied manually into firewall rules.

Social advertising platform crawler documentation is less centralized and less operationally explicit than Google’s crawler documentation. The safest transferable recommendation is architectural: do not put public share previews, ad previews, or ad destinations behind challenge pages, sign-in walls, brittle user-agent-only blocks, or geo/session-dependent rendering. If a social preview bot is business-critical, verify it with both user-agent and provider source signals before carving an exception.

Methodology and sources

This article is based on a review of the source research provided for publication and the official documentation it references from Cloudflare, Google Search Central, Google Ads, Google Merchant Center, OpenAI, Anthropic, and Perplexity. The review focuses on operational configuration decisions for SEO and web marketing teams: crawler reachability, ad delivery diagnostics, Merchant Center product validation, AI bot policy, cache behavior, redirects, SSL/TLS, and monitoring.

• Cloudflare DNS proxy status documentation
• Cloudflare SSL/TLS get started documentation
• Cloudflare WAF custom rules documentation
• Cloudflare WAF Skip action documentation
• Cloudflare IP Access Rules documentation
• Cloudflare rate limiting rules documentation
• Cloudflare challenge concepts
• Cloudflare JavaScript Detections documentation
• Cloudflare verified bots documentation
• Cloudflare managed AI robots.txt documentation
• Cloudflare cache behavior documentation
• Cloudflare bypass cache on cookie example
• Cloudflare Workers and cache interaction documentation
• Cloudflare CORS and cache documentation
• Google Ads landing page requirements and troubleshooting
• Google common crawlers documentation
• Google special-case crawlers documentation
• Google crawler verification documentation
• Google robots.txt specification notes
• Google Search Central robots.txt introduction
• Google Search Central robots meta tag and X-Robots-Tag documentation
• Google Search Central canonicalization documentation
• Google Search Central redirects documentation
• Google crawl budget documentation
• Google Search Central localized versions and hreflang documentation
• Google Merchant Center structured data documentation
• Google Merchant Center crawler and StoreBot guidance
• OpenAI crawler and user agent documentation
• OpenAI publishers and developers FAQ
• OpenAI ChatGPT-User IP JSON
• Perplexity crawler documentation
• PerplexityBot IP JSON
• Anthropic crawler and blocking documentation
• Anthropic bots JSON

This article is for technical and operational information only. metricfixer is not affiliated with Cloudflare, Google, OpenAI, Anthropic, Perplexity, or other third-party platforms mentioned here. Search-engine, advertising-platform, CDN, and AI-crawler behavior may change after publication, and production rules should be tested against the exact website, Cloudflare plan, application stack, and business policy.