--- title: Should You Block AI Crawlers? B2B Trade-Offs description: Should you block AI crawlers? A B2B decision guide to training vs search bots, what blocking protects and costs, and rules for choosing what to allow. image: https://www.finemediabw.com/hubfs/Blog/AEO/Decision/Should%20You%20Block%20AI%20Crawlers%20-%20The%20Trade-Offs%20for%20B2B%20Sites/block-ai-crawlers-share-1200x630.png --- [![Fine\_Media\_logo\_Set\_HubSpot Website Logo Stacked](https://www.finemediabw.com/hubfs/Fine_Media_logo_Set_HubSpot%20Website%20Logo%20Stacked.svg "Fine_Media_logo_Set_HubSpot Website Logo Stacked")](https://www.finemediabw.com/) - Solutions - [Inbound Marketing](https://www.finemediabw.com/inbound-marketing) - [Website Design](https://www.finemediabw.com/website-design-and-development) - [Sales Enablement](https://www.finemediabw.com/sales-enablement) - [Customer Success](https://www.finemediabw.com/customer-success) - [HubSpot Services](https://www.finemediabw.com/hubspot-services) - [Insights](https://www.finemediabw.com/blog) - [About](https://www.finemediabw.com/about-us) Show submenu for About - [About us](https://www.finemediabw.com/about-us) - [Meet the Team](https://www.finemediabw.com/our-team) - [Our Story](https://www.finemediabw.com/our-story) - Solutions - [Inbound Marketing](https://www.finemediabw.com/inbound-marketing) - [Website Design](https://www.finemediabw.com/website-design-and-development) - [Sales Enablement](https://www.finemediabw.com/sales-enablement) - [Customer Success](https://www.finemediabw.com/customer-success) - [HubSpot Services](https://www.finemediabw.com/hubspot-services) - [Insights](https://www.finemediabw.com/blog) - [About](https://www.finemediabw.com/about-us) Show submenu for About - [About us](https://www.finemediabw.com/about-us) - [Meet the Team](https://www.finemediabw.com/our-team) - [Our Story](https://www.finemediabw.com/our-story) - [Contact us](https://www.finemediabw.com/contact) - [Contact us](https://www.finemediabw.com/contact) ![Should You Block AI Crawlers? The Trade-Offs for B2B Sites. White and steel-blue type on a deep blue-grey ground, Propello.](https://www.finemediabw.com/hubfs/Blog/AEO/Decision/Should%20You%20Block%20AI%20Crawlers%20-%20The%20Trade-Offs%20for%20B2B%20Sites/block-ai-crawlers-share-1200x630.png) Oct 11, 2026, 9:34:28 PM | [AEO & AI Search](https://www.finemediabw.com/blog/tag/aeo-ai-search) # Should You Block AI Crawlers? The Trade-Offs for B2B Sites Should you block AI crawlers? A B2B decision guide to training vs search bots, what blocking protects and costs, and rules for choosing what to allow. Share - [mailto:?&subject=Should%20You%20Block%20AI%20Crawlers?%20The%20Trade-Offs%20for%20B2B%20Sites&body=Should%20You%20Block%20AI%20Crawlers?%20The%20Trade-Offs%20for%20B2B%20Sites%0A(https%3A%2F%2Fwww.finemediabw.com%2Fblog%2Fblock-ai-crawlers)](mailto:?&subject=Should%20You%20Block%20AI%20Crawlers?%20The%20Trade-Offs%20for%20B2B%20Sites&body=Should%20You%20Block%20AI%20Crawlers?%20The%20Trade-Offs%20for%20B2B%20Sites%0A(https%3A%2F%2Fwww.finemediabw.com%2Fblog%2Fblock-ai-crawlers)) - - - Most B2B sites should not block AI crawlers wholesale. Allow the crawlers that let AI search engines find and cite your pages, decide separately about the crawlers that collect content for training, and protect anything truly confidential with a login instead of a robots.txt rule. This article is for the CEOs, founders, CROs and marketing leaders who have been asked whether to block AI bots. You will get a plain definition, what blocking protects and what it costs, page-level alternatives to a full block, and decision rules you can apply by Friday.   | **AI Crawler** An AI crawler is an automated program that visits web pages on behalf of an AI company, to collect content for training, to build an AI search index, or to fetch a page for a user. | | --- | ## Why AI crawlers differ from the web crawlers you already allow Artificial intelligence tools now answer questions that buyers once typed into search engines. Search engines have long sent web crawlers to read websites on the open internet, and the trade was simple: they index your pages and send visitors. AI crawlers change that trade. Some collect text so an AI developer can teach a large language model, an AI model that learns patterns from text, and the benefit to the website owners who supplied it is indirect. Others build the index behind an AI search engine, which can cite you directly as the original source of an answer. Each of the AI companies publishes its own list of bots, and the lists do not match, so one rule rarely covers them all. Website owners can spot visits in their logs by the user agent each request reports, but Microsoft's [guidance on its crawlers](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) warns that a user agent string is easy to fake, so confirm a visit against the operator's published IP ranges. ### How an AI chatbot or AI agent fetches a page for a person When someone asks an AI chatbot a question that needs fresh information, the AI chatbot may fetch a page on the spot. AI agents that work through complex tasks for a person do the same. These fetches follow different rules from scheduled crawls, which is why the table below separates them. There is a reason to care what they read. OpenAI's [help article on web search in ChatGPT](https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt) says results and citations can be incomplete, outdated or incorrect. Inaccurate information about your offer can reach a buyer, so the pages these tools read should be current and plain. ### What AI crawlers collect, and how engines use it AI crawlers are not limited to text. Search engines run separate crawlers for images and video, and Google documents Googlebot-Image and Googlebot-Video among its crawlers. What an AI search engine does with the material is more than keyword matching: OpenAI says ChatGPT search rewrites a question into one or more targeted queries, so the words in a buyer's prompt are not always the words the engine searches. Perplexity's [help center](https://www.perplexity.ai/help-center/en/articles/10352895-how-does-perplexity-work) says its answer engine goes beyond simple keyword searches, uses language models to understand the context of a question, and searches the web in real time. Systems built on machine learning and natural language processing can also adapt to changing page layouts, read loosely structured pages and process large volumes automatically, which makes real-time monitoring of web content practical. Either way, a clean, plain page structure helps. Reach matters here. OpenAI says web search is available in the free version on its free plan as well as paid ones, within each plan's usage limits, and Perplexity says free users get limited access to its Pro Search each day while subscribers get more. Perplexity's Deep Research feature performs dozens of searches and reads hundreds of sources for a single report, so one buyer question can touch many websites at once. Because user-requested fetches happen at the moment of the question, a page you corrected this morning can be what an AI agent reads this afternoon. That is why validating your own pages matters as much as the crawler rule. ## Blocking a training crawler and blocking a search crawler cost different things ![Three columns compared on three rows, with the AI search index column in the dark. Training crawlers, search index crawlers and user-requested fetches, each with example bots, what blocking does and what it costs.](https://www.finemediabw.com/hs-fs/hubfs/Blog/AEO/Decision/Should%20You%20Block%20AI%20Crawlers%20-%20The%20Trade-Offs%20for%20B2B%20Sites/block-crawlers-crawler-jobs-compared-blog-1600x900.png?width=1600&height=900&name=block-crawlers-crawler-jobs-compared-blog-1600x900.png) A single "block AI" decision hides three separate jobs. Every major operator documents them separately, and each lets you set a rule per crawler. Treat them as three decisions, because the costs are not the same. The first job is collecting content that may be used to train a large language model. The second is building the index an AI search engine draws on when it answers a buyer. The third is fetching one page because a person asked an AI chatbot a question that needs it. | Job | Examples from the operators | If you block it | What it costs | | --- | --- | --- | --- | | Training | GPTBot, ClaudeBot, Google-Extended | Your content is signaled as out of bounds for training future models | Models may know less about your category in your words | | AI search index | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot | Your pages stop being surfaced in that engine's answers | Direct loss of visibility where buyers research | | User-requested fetch | ChatGPT-User, Claude-User, Perplexity-User | Mixed: depends on the operator | May reduce visibility for user-directed search | OpenAI says each of its settings is independent, so a site can appear in ChatGPT search while declining training. For the mechanics of the training crawler itself, see our explainer on [what GPTBot is and how AI crawlers read your site](https://www.finemediabw.com/blog/gptbot). This article stays on the decision. ### What the search bots decide for your visibility OAI-SearchBot is the OpenAI crawler that indexes pages for ChatGPT search results. OpenAI's [crawler documentation](https://developers.openai.com/api/docs/bots) says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. It recommends allowing the bot, and its published IP ranges, to help a site appear in search results. Anthropic's [help center page on its crawlers](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) says disabling Claude-SearchBot prevents indexing for search and may reduce your visibility and accuracy in results. Perplexity's [bot documentation](https://docs.perplexity.ai/guides/bots) recommends allowing PerplexityBot so your site appears in its results. ### How much referral traffic training crawlers return The business question for a training crawler is what you get back. The figures below come from Cloudflare Radar, Cloudflare's public view of traffic on the internet. [Cloudflare's August 2025 analysis of traffic on its own network](https://blog.cloudflare.com/crawlers-click-ai-bots-training/) found that training accounted for 79% of AI crawling in July 2025, up from 72% a year earlier. The same post reports that OpenAI's crawlers visited 1,091 pages for every visitor it referred back that month. Cloudflare measures this across its network and classifies purpose from operator disclosures, so read it as a pattern, not a forecast for your site. It does suggest that blocking a training crawler gives up little direct traffic, while blocking a search crawler gives up the part that can send buyers. ## What blocking protects, and what it does not ![Three stage cards, fetched by a person, named by others and left public, with the six places a crawler block leaks listed under the stage where each happens.](https://www.finemediabw.com/hs-fs/hubfs/Blog/AEO/Decision/Should%20You%20Block%20AI%20Crawlers%20-%20The%20Trade-Offs%20for%20B2B%20Sites/block-crawlers-block-protects-limits-blog-1600x900.png?width=1600&height=900&name=block-crawlers-block-protects-limits-blog-1600x900.png) Blocking protects one thing reliably: it tells a compliant operator that your content should stay out of AI model training. OpenAI says disallowing GPTBot indicates a site's content should not be used in training its models. Anthropic says restricting ClaudeBot signals that your future materials should be excluded from its training datasets. Think of the rule as three questions about access: which web crawlers may read your websites, for what purpose, and with what proof. Access for search bots brings visibility, access for training bots returns little traffic, and access for user-requested fetches is mostly outside your control. Notice the word "future" and the word "signal." A robots.txt rule is a request that reputable operators honor, not a lock on the door. ### Where a robots.txt block stops working User-requested fetches are the gap. OpenAI says that because ChatGPT-User actions are initiated by a person, robots.txt rules may not apply. Perplexity says Perplexity-User generally ignores robots.txt for the same reason. Anthropic describes a different approach: Claude-User lets site owners control which sites it can reach. Google's own [introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro) makes the same point about every file of this kind: it cannot enforce crawler behavior, respectable web crawlers obey it and others might not, so private material belongs behind a password. Robots.txt also does nothing about malicious bots, which ignore it by design. If a crawler's load is a problem, your technical teams should rate limit it at the server or firewall. Anthropic advises against blocking by IP address as an opt-out, because that may impede its ability to read your robots.txt file. ### Check what your robots.txt file says today Every website keeps a robots.txt file at its root, and any web browser can open it as plain text. Open yourdomain.com/robots.txt and read it. The file follows the Robots Exclusion Protocol, and each group starts with a user agent line that names the crawler it addresses. Look for user agent lines naming GPTBot, ClaudeBot or Google-Extended, and any other AI bots, and for a wildcard line that addresses every crawler at once. If you run multiple sites or subdomains, Anthropic says to add its rule to every subdomain you want to opt out. Then look at your page templates for robots meta tags, because a stray noindex or noarchive tag can limit AI answers as well as search listings. Server logs and a bot directory are the tools that confirm what each crawler can access, so website owners should look at the file and the logs together. ### A block may not stop a citation Blocking a crawler does not guarantee your name stays out of an answer. OpenAI says a site opted out of OAI-SearchBot can still appear as a navigational link. Google's robots.txt introduction says a blocked page can still be indexed if other sites link to it, and AI answers can repeat inaccurate information that others publish about you. A crawler rule controls what an operator fetches from you, not what others say about you, so keep your own original content accurate and easy to find on the internet. ### What you cannot protect with any crawler rule Anything a visitor can open without logging in is public, and a crawler rule does not change that. If your pricing research, benchmark data or training library is the product, put it behind authentication. Use crawler rules to state preferences about content you publish openly, not to guard secrets. ## Page-level controls protect one section without blocking the site A full block is a blunt tool, and teams that block crawlers by habit often give up more than they meant to. Search engines give you finer controls, and for a B2B site with a mix of public marketing pages and premium material, they are often the better answer. Google says its robots.txt directives for Googlebot are the control for how your site is crawled for Search, and that AI is built into Search. To limit what is shown from your pages, it points to nosnippet, data-nosnippet, max-snippet and noindex. Its Google-Extended token is separate: it manages training and grounding for Gemini Apps and Vertex AI, and Google says it does not affect inclusion in Google Search. Google calls Google AI Overviews and AI Mode its AI features. Details are on Google's [common crawlers page](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) and its page on [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features). For Bing, and for Microsoft Copilot users, Microsoft documents the robots meta tags in its [Bing Webmaster guidance](https://www.bing.com/webmasters/help/robots-meta-tags-and-attributes-that-bing-supports-5198d240). The noarchive tag tells Bing not to link a page in Chat and Copilot. The nocache tag limits Chat and Copilot to showing a URL, title and snippet. Both can be set on a single page or sent as an X-Robots-Tag header. Robots meta tags can keep AI answers away from specific parts of a site, and data-nosnippet does the same for a section within a page. They let you keep a gated report out of AI answers while the pages that win buyers stay open. Google adds one more reassurance about its AI features: there is no extra technical requirement for Google AI Overviews or AI Mode, and no need to create new machine-readable files or AI text files. Our guide to [llms.txt](https://www.finemediabw.com/blog/llms-txt) covers why that file is not a control either. ## Four rules for deciding which AI crawlers to block ![Four columns compared on three rows, with the unsure column in the dark. Sites that exist to be found, sites that sell the content, sites with contract limits and unsure sites, each with what to do about search crawlers, training crawlers and one further step.](https://www.finemediabw.com/hs-fs/hubfs/Blog/AEO/Decision/Should%20You%20Block%20AI%20Crawlers%20-%20The%20Trade-Offs%20for%20B2B%20Sites/block-crawlers-block-decision-rules-blog-1600x900.png?width=1600&height=900&name=block-crawlers-block-decision-rules-blog-1600x900.png) These rules are our reading of the operators' documentation, not a claim any operator makes. They assume you sell to businesses and that buyers research before they talk to you. ### If your site exists to be found, allow the search crawlers For most B2B sites, marketing pages, articles and product explanations exist so buyers find you. Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot and Bingbot. If you also want to appear in Google, Googlebot must be allowed, which you almost certainly already do. ### If you sell the content itself, block training and gate the rest Publishers whose original content is paid research, data or courses, and teams for whom content creation is a real expense, have a clear reason to decline training crawlers. Disallow GPTBot, ClaudeBot and, if you wish, Google-Extended. Then move the valuable material behind a login, because robots.txt alone will not hold it. ### If a contract or regulation limits reuse, ask legal first Some client agreements and privacy obligations restrict how published material may be reused. A crawler policy is also a risk policy, because legal compliance and data ownership depend on what you publish and what your agreements allow. Your counsel should confirm what applies, and the decision should be recorded with a date and an owner. ### If you are unsure, allow search, decline training and review in a quarter It is better to block crawlers on purpose than by habit. This is the lowest-regret position for a growth-stage company. You keep the visibility that can send buyers, you state a preference on training, and you can change a single file later. Artificial intelligence companies add and rename crawlers, so a scheduled review matters more than a perfect first answer. ## How the decision changes from early stage to scale These are illustrative stages, not client results. The same four rules lead to different settings as a company grows. ### Early stage: a small site that exists to be found Allow the search crawlers and consider allowing training too, since a young company wants AI models to describe its category correctly. Block nothing by default. Review the robots.txt file each quarter, and keep the customer portal behind a login. ### Growth stage: a content library that brings in buyers Allow the search crawlers on the public library and decide on training by content type. If a paid research section appears, disallow training crawlers for it and keep it behind authentication. Add the noarchive robots meta tag to preview pages that should not be linked in Microsoft Copilot answers. Review the file and the logs every month. ### Scale stage: gated research, several regions and a legal team Public marketing pages stay open to search crawlers. Gated research is blocked for training and sits behind a login. Anything that touches client work is not published at all. Legal signs off on the policy once, engineering records it for every region and subdomain, and a named owner reviews it each quarter and whenever an operator adds a crawler. ## What goes wrong when the crawler decision is left to chance Three failures show up again and again. Each is quiet, and each costs visibility before anyone notices. The first is a blocked search crawler. A blanket rule written to stop training bots can also stop OAI-SearchBot, Claude-SearchBot or PerplexityBot, and the operators say that means your pages are not shown in their answers. The second is an inherited firewall or content delivery network rule that blocks unfamiliar bots. Perplexity's documentation says a web application firewall may need to allow its bots explicitly, combining user agent and published IP range, and OpenAI asks that its published IP ranges be allowed. The third is a stale robots.txt file that nobody owns. Operators add and rename crawlers, Google ignores content after 500 kibibytes according to its [robots.txt specification page](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt), and a file that grew by habit can say the opposite of the policy. Use this short checklist with your web team: - Confirm the robots.txt file on every website and subdomain names each crawler you made a decision about. - Confirm the robots meta tags on gated pages say what you intend. - Confirm your firewall and content delivery network give access to the search crawlers you allow. - Use log tools to watch for new AI bots and unfamiliar user agents each month. - Record the decision, the owner and the review date, and keep a copy of the file and the log data. Search your server logs for the user agent string of each crawler you meant to allow. Cloudflare's [Radar bots directory](https://radar.cloudflare.com/bots/directory) is a public list of known bots that helps your team recognize what it finds. The Crawl-delay setting is not part of the robots.txt standard, but Anthropic says it supports it if you want to slow a crawler instead of stopping it. Both OpenAI and Perplexity say a robots.txt update can take about 24 hours to take effect, so test the day after you change it. For how engines choose what to cite once they can reach you, see our guide to [AI search ranking factors](https://www.finemediabw.com/blog/ai-search-ranking-factors). ## What you gain when this is done properly Artificial intelligence keeps changing where buyers research, so a deliberate crawler policy gives each team something it can use, and it ends ad hoc blocking decisions made after a complaint. ### Sales meets buyers who already know your category When search crawlers can read your pages, AI assistants can answer buyer questions from your material. Prospects who asked an AI chatbot about your category arrive with your terms in mind, so reps spend less of a first call correcting basics, whether the buyer used ChatGPT, Perplexity or Microsoft Copilot. ### Marketing protects premium assets and keeps findable content open Marketing gets a written rule for what is meant to be found and what is protected. Gated reports stay gated, public explainers stay open, and nobody has to relitigate the choice every time a new bot appears. ### Customer success sees fewer wrong answers about the product Customers ask an AI chatbot how to use your product, and AI tools answer from the pages they can reach. If your help content is open to the right crawlers, the answers draw on your current instructions, and fewer tickets start with a wrong answer. ## Decide once, write it down and review it Blocking AI crawlers is not one decision. Allow the crawlers that let AI search engines find and cite you, make a separate and reversible call on training, and use logins and page-level tags for what must stay private. Propello designs and builds connected go-to-market systems on HubSpot. If you want your content access and AI search visibility planned as part of one revenue system alongside your other business systems, an audit is the place to start. [Book a Propello GTM Audit](https://www.finemediabw.com/contact) ## Frequently asked questions Should a B2B company block AI crawlers? Usually not entirely. Allow the search crawlers so an AI chatbot or answer engine can cite your pages, and decide separately whether to allow training crawlers. Block training if your content is the product or a contract restricts reuse. Keep confidential material behind a login, not behind a robots.txt rule. Does blocking AI crawlers hurt my Google rankings? Blocking a training token such as Google-Extended does not affect inclusion in Google Search, according to Google. Blocking GPTBot or any other non-Google crawler does not change how Googlebot crawls you. Blocking Googlebot itself is different, because that is the control for how Search crawls your site. Does robots.txt still work for AI crawlers? For automatic crawlers from the main operators, yes. OpenAI, Anthropic and Perplexity all document robots.txt rules for their bots. It does not bind malicious bots, and OpenAI and Perplexity say user-requested fetches may ignore it. Treat it as a policy signal, not a security control. How long does a robots.txt change take to apply? OpenAI says its systems can take about 24 hours to adjust to a robots.txt update, and Perplexity says up to 24 hours. Plan to verify the result the next day, and check your firewall or content delivery network as well, since it can block a crawler your file allows. Who should own the decision on blocking AI bots? A revenue or marketing leader should own the policy, with legal and IT staff as reviewers. The choice affects where buyers find you, what you protect and what contracts allow. Record the decision with a date and review it every quarter, since operators change their crawlers. ![Tumisang Bogwasi](https://app.hubspot.com/settings/avatar/77d7e2eaad8ff71b24463dcc39a31e9e) ### Written By: Tumisang Bogwasi Tumisang Bogwasi is the founder and CEO of Propello, a HubSpot partner that designs and builds connected go-to-market systems. [mailto:tumib@finemediabw.com](mailto:tumib@finemediabw.com) ## You May Also Like [Learn More](https://www.example.com) [![Fine\_Media\_logo\_Set\_HubSpot Website Logo Stacked White BG White-1](https://www.finemediabw.com/hubfs/Fine_Media_logo_Set_HubSpot%20Website%20Logo%20Stacked%20White%20BG%20White-1.svg "Fine_Media_logo_Set_HubSpot Website Logo Stacked White BG White-1")](https://www.finemediabw.com) We help you propel growth at every stage of the customer journey with inbound marketing and Hubspot. ![provider-horizontal-white](https://www.finemediabw.com/hubfs/provider-horizontal-white.svg "provider-horizontal-white") Blogs [What is Inbound Marketing](https://www.finemediabw.com/blog/what-is-inbound-marketing) [Why work with a HubSpot Agency](https://www.finemediabw.com/blog/why-work-with-hubspot-agency) [Benefits of a Website](https://www.finemediabw.com/blog/the-power-of-website-design-a-cornerstone-of-brand-identity) [Ultimate Guide to Closing more Deals](https://www.finemediabw.com/blog/how-to-close-more-deals-with-sales-enablement) [Why Every Business Needs Customer Success](https://www.finemediabw.com/blog/why-customer-success-is-important) Services   [Inbound Marketing](https://www.finemediabw.com/inbound-marketing) [Website Design](https://www.finemediabw.com/website-design-and-development) [Sales Enablement](https://www.finemediabw.com/sales-enablement) [Customer Success](https://www.finemediabw.com/customer-success) [HubSpot Services](https://www.finemediabw.com/hubspot-services) Quick Links   [Home](https://www.finemediabw.com/) Services [About us](https://www.finemediabw.com/about-us) [Contact](https://www.finemediabw.com/contact-us) © 2026 Fine Media (Pty) Ltd. All rights reserved. [Legal Notice](https://www.finemediabw.com/user-agreement) | [Privacy Policy](https://www.finemediabw.com/privacy-policy)  ```json { "@context" : "https://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "Tumisang Bogwasi", "url" : "https://www.finemediabw.com/blog/author/tumisang-bogwasi" }, "datePublished" : "2026-10-11T19:34:28.000Z", "headline" : "Should You Block AI Crawlers? B2B Trade-Offs", "image" : [ "https://www.finemediabw.com/hubfs/Blog/AEO/Decision/Should%20You%20Block%20AI%20Crawlers%20-%20The%20Trade-Offs%20for%20B2B%20Sites/block-ai-crawlers-share-1200x630.png" ], "mainEntityOfPage" : { "@id" : "https://www.finemediabw.com/blog/block-ai-crawlers", "@type" : "WebPage" }, "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "url" : "https://www.finemediabw.com/hubfs/Fine_Media_logo_Set%20copy_HubSpot_Logo_Long_Stacked.svg" }, "name" : "Fine Media" } } ``` ```json { "@context" : "https://schema.org", "@type" : "FAQPage", "mainEntity" : [ { "@type" : "Question", "acceptedAnswer" : { "@type" : "Answer", "text" : "Usually not entirely. Allow the search crawlers so an AI chatbot or answer engine can cite your pages, and decide separately whether to allow training crawlers. Block training if your content is the product or a contract restricts reuse. Keep confidential material behind a login, not behind a robots.txt rule." }, "name" : "Should a B2B company block AI crawlers?" }, { "@type" : "Question", "acceptedAnswer" : { "@type" : "Answer", "text" : "Blocking a training token such as Google-Extended does not affect inclusion in Google Search, according to Google. Blocking GPTBot or any other non-Google crawler does not change how Googlebot crawls you. Blocking Googlebot itself is different, because that is the control for how Search crawls your site." }, "name" : "Does blocking AI crawlers hurt my Google rankings?" }, { "@type" : "Question", "acceptedAnswer" : { "@type" : "Answer", "text" : "For automatic crawlers from the main operators, yes. OpenAI, Anthropic and Perplexity all document robots.txt rules for their bots. It does not bind malicious bots, and OpenAI and Perplexity say user-requested fetches may ignore it. Treat it as a policy signal, not a security control." }, "name" : "Does robots.txt still work for AI crawlers?" }, { "@type" : "Question", "acceptedAnswer" : { "@type" : "Answer", "text" : "OpenAI says its systems can take about 24 hours to adjust to a robots.txt update, and Perplexity says up to 24 hours. Plan to verify the result the next day, and check your firewall or content delivery network as well, since it can block a crawler your file allows." }, "name" : "How long does a robots.txt change take to apply?" }, { "@type" : "Question", "acceptedAnswer" : { "@type" : "Answer", "text" : "A revenue or marketing leader should own the policy, with legal and IT staff as reviewers. The choice affects where buyers find you, what you protect and what contracts allow. Record the decision with a date and review it every quarter, since operators change their crawlers." }, "name" : "Who should own the decision on blocking AI bots?" } ] } ```