--- title: What Is llms.txt and Do You Need One? description: What is llms.txt? Learn what an llms.txt file contains, what Google and other AI engines say about it, and whether your B2B site actually needs one. image: https://www.finemediabw.com/hubfs/Blog/AEO/Awareness/What%20Is%20llms.txt%20and%20Do%20You%20Need%20One/llms-txt-share-1200x630.png --- [![Fine\_Media\_logo\_Set\_HubSpot Website Logo Stacked](https://www.finemediabw.com/hubfs/Fine_Media_logo_Set_HubSpot%20Website%20Logo%20Stacked.svg "Fine_Media_logo_Set_HubSpot Website Logo Stacked")](https://www.finemediabw.com/) - Solutions - [Inbound Marketing](https://www.finemediabw.com/inbound-marketing) - [Website Design](https://www.finemediabw.com/website-design-and-development) - [Sales Enablement](https://www.finemediabw.com/sales-enablement) - [Customer Success](https://www.finemediabw.com/customer-success) - [HubSpot Services](https://www.finemediabw.com/hubspot-services) - [Insights](https://www.finemediabw.com/blog) - [About](https://www.finemediabw.com/about-us) Show submenu for About - [About us](https://www.finemediabw.com/about-us) - [Meet the Team](https://www.finemediabw.com/our-team) - [Our Story](https://www.finemediabw.com/our-story) - Solutions - [Inbound Marketing](https://www.finemediabw.com/inbound-marketing) - [Website Design](https://www.finemediabw.com/website-design-and-development) - [Sales Enablement](https://www.finemediabw.com/sales-enablement) - [Customer Success](https://www.finemediabw.com/customer-success) - [HubSpot Services](https://www.finemediabw.com/hubspot-services) - [Insights](https://www.finemediabw.com/blog) - [About](https://www.finemediabw.com/about-us) Show submenu for About - [About us](https://www.finemediabw.com/about-us) - [Meet the Team](https://www.finemediabw.com/our-team) - [Our Story](https://www.finemediabw.com/our-story) - [Contact us](https://www.finemediabw.com/contact) - [Contact us](https://www.finemediabw.com/contact) ![What Is llms.txt and Do You Need One? Dark type on a pale blue-grey ground, Propello.](https://www.finemediabw.com/hubfs/Blog/AEO/Awareness/What%20Is%20llms.txt%20and%20Do%20You%20Need%20One/llms-txt-share-1200x630.png) Oct 10, 2026, 1:36:13 PM | [AEO & AI Search](https://www.finemediabw.com/blog/tag/aeo-ai-search) # What Is llms.txt and Do You Need One? What is llms.txt? Learn what an llms.txt file contains, what Google and other AI engines say about it, and whether your B2B site actually needs one. Share - [mailto:?&subject=What%20Is%20llms.txt%20and%20Do%20You%20Need%20One?&body=What%20Is%20llms.txt%20and%20Do%20You%20Need%20One?%0A(https%3A%2F%2Fwww.finemediabw.com%2Fblog%2Fllms-txt)](mailto:?&subject=What%20Is%20llms.txt%20and%20Do%20You%20Need%20One?&body=What%20Is%20llms.txt%20and%20Do%20You%20Need%20One?%0A(https%3A%2F%2Fwww.finemediabw.com%2Fblog%2Fllms-txt)) - - - If you lead revenue at a B2B company, someone has probably told you to add an llms.txt file to show up in AI answers. The short answer: it is a small, optional file, no major AI platform has publicly committed to reading it for answers, and it is not where AI visibility is won. This article is for CEOs, founders and revenue leaders deciding where to spend limited effort. You will learn what the file contains, how it differs from robots.txt, what Google, OpenAI, Perplexity and Anthropic say in their own documentation, what the evidence shows, and when a file is still worth publishing. Our explainer on [GPTBot](https://www.finemediabw.com/blog/gptbot) covers the crawlers these rules control.   | **llms.txt** llms.txt is a proposed standard for a plain text Markdown file placed on a website that gives AI systems a short summary of the site's content and links to its most useful pages. | | --- | ## What an llms.txt file contains and how AI systems are meant to use it ![An llms.txt file shown as five layers held by one spine, from the required site name up through the summary, notes, link sections and the optional section.](https://www.finemediabw.com/hs-fs/hubfs/Blog/AEO/Awareness/What%20Is%20llms.txt%20and%20Do%20You%20Need%20One/llms-txt-llms-txt-file-anatomy-blog-1600x900.png?width=1600&height=900&name=llms-txt-llms-txt-file-anatomy-blog-1600x900.png) The idea comes from Jeremy Howard, who published the proposal at [llmstxt.org](https://llmstxt.org/) on September 3, 2024. It asks site owners to add a simple file that helps AI agents and large language models understand a website without reading every page, so they can answer questions with better context. It is aimed at the language models behind assistants, not at human readers. Large language models cannot take in a whole site at once. Context windows are limited, and a typical HTML page of web content is full of menus, scripts and layout code. A short, clean file that names the most important content and gives a high level structure of the site is meant to give AI tools and AI models a better starting point. In theory it also saves them time crawling your pages, and it may help retrieval-augmented generation systems find the right pages. Those are design goals, not measured results. ### The parts of the file, in order The proposal sets out a basic structure built from key information, not from everything you publish. The file name is the lowercase llms.txt, written in Markdown, and the sections follow a set order. Only the first part is required, and the rest can be left out. - An H1 heading with the name of the site or project. This is the only required part. - A blockquote with a short summary of what the site is for. - Optional sections of detailed information in plain Markdown, without extra headings. - Optional H2 sections that hold file lists: bullet links to key pages, each with a name, a URL and a short note. - An H2 section named Optional, which by convention holds secondary links an agent can skip when it needs a shorter context. For example, here is an invented llms.txt file for a fictional company, set out in one of the code blocks you could copy, so you can see how small a plain text file it is: ``` # Example Co > Example Co sells scheduling software to dental clinics. ## Product - [Pricing](https://example.com/pricing): Plans and what each includes - [Integrations](https://example.com/integrations): Systems we connect to ## Docs - [Getting started](https://example.com/docs/start.md): Setup in one page ## Optional - [Company history](https://example.com/about): Background reading ``` ### Markdown files, llms-full.txt and where the file lives The proposal says the file belongs at /llms.txt in the root directory of your site, though it can also sit at another root path and then covers the pages under that path. When several files apply, the most specific one wins. It also suggests that pages with useful information offer a clean markdown version at the same address, with .md added to the URL. Those markdown files, sometimes called md versions of your pages, are what the links in your llms.txt file point to. You will often see a second file called llms-full.txt, which some sites use to publish their full documentation in one place. The proposal page does not define it, so treat it as a convention that some documentation platforms follow, not as part of the standard. ## How llms.txt differs from existing standards like robots.txt and sitemap.xml People confuse the three files because all of them sit in the root directory and all of them are read by software. They do different jobs, and only one of them controls access. Unlike the XML used by sitemaps, llms.txt is Markdown. | File | What it does | Who it is written for | | --- | --- | --- | | robots.txt | Tells crawlers which parts of a site they may or may not fetch | Search engines and AI crawlers | | sitemap.xml | Lists the URLs you want search engines to discover | Search engines | | llms.txt | Offers a short, curated summary and links to key content | AI agents and AI tools, if they choose to read it | The key difference is control. A robots.txt rule is a documented instruction that crawlers from OpenAI and Anthropic say they follow. An llms.txt file controls nothing. It cannot block a crawler, grant access or force an engine to use your links. Access control stays in robots.txt. A sitemap lists all the indexable pages written for human readers. An llms.txt file is a short, prioritized list of the pages that matter most, not a replacement for full documentation or advanced retrieval systems. Because the files do different jobs, blocking or allowing AI crawlers is a separate decision from publishing a summary file. If you have not made it yet, start with robots.txt and your access settings, not with llms.txt. ## Which AI engines have committed to reading llms.txt ![A table of four engines, Google Search, OpenAI, Perplexity and Anthropic, against robots.txt and llms.txt, showing that each engine states how it treats robots.txt and none has committed to reading llms.txt.](https://www.finemediabw.com/hs-fs/hubfs/Blog/AEO/Awareness/What%20Is%20llms.txt%20and%20Do%20You%20Need%20One/llms-txt-who-reads-which-file-v2-blog-1600x900.png?width=1600&height=900&name=llms-txt-who-reads-which-file-v2-blog-1600x900.png) This is the question that decides whether the file matters for you, so here is what each company says on its own pages, as of October 2026. A page that does not mention the file is not a commitment, and nothing below is inferred. ### Google Search says it ignores the file Google is the clearest on its llms.txt position. Its [generative AI optimization guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide), updated in July 2026, lists llms.txt under things you do not need to do. It says maintaining the file will neither harm nor help your visibility or rankings in Google Search, because Google Search ignores it. The same guide says it is fine to create the file for other services that use it. Google's page on [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) adds that you do not need new machine readable files or AI text files to appear in generative AI search features such as AI Overviews or AI Mode. ### Google Lighthouse treats it as optional Google also makes a developer tool, and it does mention the file. The [Google Lighthouse audit for llms.txt](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt) calls it an emerging convention and a machine-readable summary of a website, specifically designed for AI agents and language models. It lists the file among its agentic browsing audits and says providing the file is optional at the moment. A missing file, shown as a 404, marks the audit Not Applicable. That is about how smoothly browser-based agents can understand a website, not about search rankings in AI search or anywhere else. ### OpenAI, Perplexity and Anthropic document crawlers, not this file OpenAI's [crawler documentation](https://developers.openai.com/api/docs/bots) explains that OAI-SearchBot, GPTBot and ChatGPT-User each have separate settings and that robots.txt controls the first two. It does not say ChatGPT reads llms.txt for answers, even though the page points readers to its own documentation index in that format. The same pattern holds elsewhere. [Perplexity's bot guide](https://docs.perplexity.ai/guides/bots) recommends allowing PerplexityBot in robots.txt and does not state that the file is used. [Anthropic's page on its crawlers](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) says its bots honor robots.txt and does not mention llms.txt at all. Publishing a file in that format on a company's own developer docs is not the same as telling the world its AI assistants read yours. Nothing guarantees that any AI system will read it, and in practice you find the file mostly on software documentation sites. ## What the evidence says about llms.txt and AI visibility Independent data points the same way. In November 2025, [SE Ranking analyzed nearly 300,000 domains](https://seranking.com/blog/llms-txt/) and found that just 10.13% had an llms.txt file. It found no correlation between having an llms.txt file and being cited in AI answers. The authors add that the llms.txt result is context-dependent and reflects the model and data they tested. This is one correlational study, so it does not prove the file never helps. A June 2026 [Ahrefs study of 137,210 domains](https://ahrefs.com/blog/llmstxt-study/) found that 28% published a valid llms.txt file, and its authors call that an upper bound because their analytics customers skew technical. In [the same study](https://ahrefs.com/blog/llmstxt-study/), 97% of those files received no requests at all in May 2026. A request is not proof that anything read the file, and the two studies sampled different sites, which is why their adoption numbers differ. Together they mean you should not expect an llms.txt file to lift your AI visibility on its own. Adoption is gaining traction, and thousands of sites now serve the file, but that says little about results. If you want evidence from your own website, check your server logs. Look for requests to /llms.txt from the AI crawlers you recognize. If nothing fetches it for months, you have your answer for your site. ## Do you need an llms.txt file for a B2B site? Probably not as a priority. Most websites do not need one. Because it is still a proposed standard, no engine has to honor it. For a typical B2B marketing site, an llms.txt file is optional, low effort, and unlikely to change how AI search engines describe you. What moves AI visibility is the work in our [guide to AI Engine Optimization (AEO)](https://www.finemediabw.com/blog/ultimate-guide-ai-engine-optimization): clear entities, answerable pages and open access for the crawlers you want. ### When it is worth the effort An llms.txt file is more useful when software, not buyers, reads your content. These are the cases where it earns a place on the list: - You publish developer documentation or API references that AI coding assistants and AI agents will be asked about. - You run a documentation platform that can generate and update the file automatically. - Your product is used through AI tools, and a clean markdown version of key pages helps them answer correctly. - You want to pass the Lighthouse audit for agentic browsing. A clear blog post or pricing page that answers one buyer question is content engines can already read, with or without a summary file. If none of these cases fits, spend the hour on the items in our [annotated AEO checklist](https://www.finemediabw.com/blog/aeo-checklist-annotated-page) and the steps in our guide on [how to optimize content for AI search](https://www.finemediabw.com/blog/how-to-optimize-content-for-ai-search). ### How to create one in an afternoon If you decide to publish an llms.txt file, keep it short and treat it like any other page you maintain on your website. 1. Pick the ten to twenty important pages, your high value content, that answer the questions buyers and customers ask most. 2. Write the H1 and a short description of what your company does and for whom, in two sentences. 3. Group the links under a few H2 headings, each with a one-line note. 4. Publish the llms.txt file as plain text in the correct directory, your site's root directory, and open it in a browser to confirm it loads. 5. Review it when your product, pricing or key pages change, so the links never go stale. ## What you gain when this is done properly ![Clarity at the centre with three arcs for what sales, marketing and customer success each gain from deciding on purpose whether to publish an llms.txt file.](https://www.finemediabw.com/hs-fs/hubfs/Blog/AEO/Awareness/What%20Is%20llms.txt%20and%20Do%20You%20Need%20One/llms-txt-what-each-team-gains-blog-1600x900.png?width=1600&height=900&name=llms-txt-what-each-team-gains-blog-1600x900.png) The benefit of an llms.txt decision is clarity about where AI visibility effort goes. Whether you publish the llms.txt file or skip it, the decision should be deliberate and written down. ### Sales hears one consistent story When you decide which pages describe your company, product and pricing, those pages become the reference. Reps and AI tools then work from the same description, and prospects who arrive after asking an assistant meet fewer surprises. ### Marketing spends effort where it counts A clear view of what an llms.txt file can and cannot do keeps your team from chasing a tactic with no documented support. The time goes to answerable content, entity consistency and crawler access instead, for example a clear blog post that answers one buyer question. ### Customer success gets cleaner help content If you publish developer docs or help articles, a tidy summary file and markdown versions turn that content into structured content that is easier for AI tools to read, with clear structured information to draw on. Customers who ask AI assistants how to configure your product get answers closer to your documentation. ## Treat llms.txt as one small part of a go-to-market system An llms.txt file is a cheap addition on top of a site that already works. It does not replace crawl access, accurate pages or the connection between AI visibility and pipeline. Think of it as housekeeping alongside traditional SEO, which still decides whether search bots and AI bots can reach your pages. Propello designs and builds connected go-to-market systems on HubSpot. If you want to know where AI search effort belongs in yours, an audit is the place to start. [Book a Propello GTM Audit](https://www.finemediabw.com/contact) ## Frequently asked questions Is llms.txt mandatory? No. It is a proposal, not a requirement. No search engine requires it to index or rank your pages, and Google's documentation says its Search ignores it. Lighthouse calls providing the file optional, and a missing file does not fail that audit. Does llms.txt really work? Nothing in the documentation we reviewed shows that Google, OpenAI, Perplexity or Anthropic use it to choose answers. SE Ranking found no correlation between having the file and being cited. It may help some agents that read it, but treat any benefit as unproven. Where do you put an llms.txt file? Put it in the root directory of your site, so it loads at yourdomain.com/llms.txt. The proposal also allows a file at another path, which then covers the pages under that path. Publish it as plain text and confirm it opens in a browser. How do you write an llms.txt file? Start with an H1 naming your company, then add a blockquote summary. Add H2 sections with Markdown bullet links to your key pages, each with a short note. Put secondary links under a section named Optional. Keep it concise and update it when pages change. What is the difference between llms.txt and robots.txt? robots.txt tells crawlers what they may fetch, and major AI crawlers document that they follow it. llms.txt offers a curated summary and links, and controls nothing. Use robots.txt to decide access, and treat llms.txt as an optional guide. ![Tumisang Bogwasi](https://app.hubspot.com/settings/avatar/77d7e2eaad8ff71b24463dcc39a31e9e) ### Written By: Tumisang Bogwasi Tumisang Bogwasi is the founder and CEO of Propello, a HubSpot partner that designs and builds connected go-to-market systems. [mailto:tumib@finemediabw.com](mailto:tumib@finemediabw.com) ## You May Also Like ### [![What Is GPTBot? How AI Crawlers Read Your Site. Dark and slate type on a white ground, Propello.](https://www.finemediabw.com/hubfs/Blog/AEO/Awareness/What%20Is%20GPTBot%20-%20How%20AI%20Crawlers%20Read%20Your%20Site/gptbot-share-1200x630.png) *Oct 10, 2026, 1:36:08 PM | AEO & AI Search* What Is GPTBot? How AI Crawlers Read Your Site](https://www.finemediabw.com/blog/gptbot) ### [![AEO vs GEO vs LLMO: What the Terms Mean and Which One Matters. White and blue-grey type on a dark navy ground with a large Propello mark behind it, Propello.](https://www.finemediabw.com/hubfs/Blog/AEO/Awareness/AEO%20vs%20GEO%20vs%20LLMO%20-%20What%20the%20Terms%20Mean%20and%20Which%20One%20Matters/aeo-vs-geo-vs-llmo-share-1200x630.png) *Oct 4, 2026, 9:30:56 AM | AEO & AI Search* AEO vs GEO vs LLMO: What the Terms Mean and Which One Matters](https://www.finemediabw.com/blog/aeo-vs-geo-vs-llmo) ### [![The Ultimate Guide to Answer Engine Optimization (AEO). Dark and slate type on a light grey ground, Propello.](https://www.finemediabw.com/hubfs/Blog/Customer%20Acquisition/Refresh/The%20Ultimate%20Guide%20to%20Answer%20Engine%20Optimization%20(AEO)/ultimate-guide-ai-engine-optimization-share-1200x630-v3.png) *Sep 17, 2025, 10:00:17 PM | AEO & AI Search* The Ultimate Guide to Answer Engine Optimization (AEO)](https://www.finemediabw.com/blog/ultimate-guide-ai-engine-optimization) [Learn More](https://www.example.com) [![Fine\_Media\_logo\_Set\_HubSpot Website Logo Stacked White BG White-1](https://www.finemediabw.com/hubfs/Fine_Media_logo_Set_HubSpot%20Website%20Logo%20Stacked%20White%20BG%20White-1.svg "Fine_Media_logo_Set_HubSpot Website Logo Stacked White BG White-1")](https://www.finemediabw.com) We help you propel growth at every stage of the customer journey with inbound marketing and Hubspot. ![provider-horizontal-white](https://www.finemediabw.com/hubfs/provider-horizontal-white.svg "provider-horizontal-white") Blogs [What is Inbound Marketing](https://www.finemediabw.com/blog/what-is-inbound-marketing) [Why work with a HubSpot Agency](https://www.finemediabw.com/blog/why-work-with-hubspot-agency) [Benefits of a Website](https://www.finemediabw.com/blog/the-power-of-website-design-a-cornerstone-of-brand-identity) [Ultimate Guide to Closing more Deals](https://www.finemediabw.com/blog/how-to-close-more-deals-with-sales-enablement) [Why Every Business Needs Customer Success](https://www.finemediabw.com/blog/why-customer-success-is-important) Services   [Inbound Marketing](https://www.finemediabw.com/inbound-marketing) [Website Design](https://www.finemediabw.com/website-design-and-development) [Sales Enablement](https://www.finemediabw.com/sales-enablement) [Customer Success](https://www.finemediabw.com/customer-success) [HubSpot Services](https://www.finemediabw.com/hubspot-services) Quick Links   [Home](https://www.finemediabw.com/) Services [About us](https://www.finemediabw.com/about-us) [Contact](https://www.finemediabw.com/contact-us) © 2026 Fine Media (Pty) Ltd. All rights reserved. [Legal Notice](https://www.finemediabw.com/user-agreement) | [Privacy Policy](https://www.finemediabw.com/privacy-policy)  ```json { "@context" : "https://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "Tumisang Bogwasi", "url" : "https://www.finemediabw.com/blog/author/tumisang-bogwasi" }, "datePublished" : "2026-10-10T11:36:13.000Z", "headline" : "What Is llms.txt and Do You Need One?", "image" : [ "https://www.finemediabw.com/hubfs/Blog/AEO/Awareness/What%20Is%20llms.txt%20and%20Do%20You%20Need%20One/llms-txt-share-1200x630.png" ], "mainEntityOfPage" : { "@id" : "https://www.finemediabw.com/blog/llms-txt", "@type" : "WebPage" }, "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "url" : "https://www.finemediabw.com/hubfs/Fine_Media_logo_Set%20copy_HubSpot_Logo_Long_Stacked.svg" }, "name" : "Fine Media" } } ``` ```json { "@context" : "https://schema.org", "@type" : "FAQPage", "mainEntity" : [ { "@type" : "Question", "acceptedAnswer" : { "@type" : "Answer", "text" : "No. It is a proposal, not a requirement. No search engine requires it to index or rank your pages, and Google's documentation says its Search ignores it. Lighthouse calls providing the file optional, and a missing file does not fail that audit." }, "name" : "Is llms.txt mandatory?" }, { "@type" : "Question", "acceptedAnswer" : { "@type" : "Answer", "text" : "Nothing in the documentation we reviewed shows that Google, OpenAI, Perplexity or Anthropic use it to choose answers. SE Ranking found no correlation between having the file and being cited. It may help some agents that read it, but treat any benefit as unproven." }, "name" : "Does llms.txt really work?" }, { "@type" : "Question", "acceptedAnswer" : { "@type" : "Answer", "text" : "Put it in the root directory of your site, so it loads at yourdomain.com/llms.txt. The proposal also allows a file at another path, which then covers the pages under that path. Publish it as plain text and confirm it opens in a browser." }, "name" : "Where do you put an llms.txt file?" }, { "@type" : "Question", "acceptedAnswer" : { "@type" : "Answer", "text" : "Start with an H1 naming your company, then add a blockquote summary. Add H2 sections with Markdown bullet links to your key pages, each with a short note. Put secondary links under a section named Optional. Keep it concise and update it when pages change." }, "name" : "How do you write an llms.txt file?" }, { "@type" : "Question", "acceptedAnswer" : { "@type" : "Answer", "text" : "robots.txt tells crawlers what they may fetch, and major AI crawlers document that they follow it. llms.txt offers a curated summary and links, and controls nothing. Use robots.txt to decide access, and treat llms.txt as an optional guide." }, "name" : "What is the difference between llms.txt and robots.txt?" } ] } ```