A practical guide to llms.txt for B2B sites — file structure, what to include, how it relates to robots.txt and JSON-LD, and an honest read of the evidence on whether AI platforms actually consume it.
In September 2024, Jeremy Howard published the llms.txt proposal: a plain text file at the root of a domain that tells language models how to interpret the site. The premise was clean. Where robots.txt tells crawlers what they may and may not index, llms.txt tells LLMs what information matters, how it is organized, and what the company behind the domain actually is.
The idea spread fast. Anthropic, Mintlify, Cursor and a long list of developer-tooling companies publish one. Two years later, most B2B sites still do not have one — and the ones that do are frequently arguing about whether it does anything at all.
That argument deserves a straight answer before the how-to, because the how-to is worthless if the premise is wrong. So this piece has two parts: what the evidence actually supports, and — for the cases where it makes sense — how to build the file properly.
First, the uncomfortable part: publishing is not parsing
Most llms.txt guides open by listing the AI companies that publish an llms.txt file, and let the reader draw the obvious conclusion: if Anthropic has one, the models must read them.
That inference does not hold, and it is worth being precise about why.
Publishing a file and consuming a file are two different behaviors. <cite index=“11-1”>Anthropic publishes its own llms.txt but has not stated that its crawlers use the standard, and Google manages AI crawl behavior through robots.txt directives with no mention of llms.txt support.</cite> <cite index=“10-1”>Google has said explicitly that the file gets no special treatment.</cite> A company publishing an llms.txt is telling you something about its documentation strategy, not about its retrieval pipeline.
No major platform has committed to parsing it. <cite index=“9-1”>There is currently no widely validated evidence that llms.txt reliably improves visibility in AI answers, citation frequency, or referral traffic.</cite> <cite index=“8-1”>Server log analysis shows GPTBot occasionally fetching llms.txt files, but not often, and the standard is not yet established across the AI industry.</cite>
The strongest observed mechanism is boring and familiar. A review of more than 1,400 llms.txt files found that <cite index=“13-1”>files ranking organically in classic search also appear as sources in Perplexity for the same query, suggesting the file is being picked up through RAG precisely because it ranks — which implies the human-readable portions of the file are what carry the weight.</cite>
Read that last one carefully, because it is the actual finding underneath the noise. The file does not open a private back channel to the model. When it works, it appears to work the same way any other indexable page works: it gets retrieved because it is findable and legible. llms.txt is subject to the ordinary rules of search, not an exemption from them.
There is a counter-position, and it comes mostly from documentation-heavy companies reporting that structured markdown summaries reduce token cost and improve retrieval quality for coding assistants. That is credible for its use case. It is also the use case the standard was designed for — developer documentation — and it does not generalize cleanly to a B2B site selling a service.
So what is the honest recommendation?
Build it, with correctly sized expectations.
The cost is roughly an hour of work and one file. The downside risk is zero. The upside is a real option on a standard that may or may not consolidate. That asymmetry justifies the file on its own — a one-hour hedge against a possible shift in how buyers find you is a reasonable trade at any size of company.
What it does not justify is sequencing. If llms.txt is the first item in your GEO audit, or if it is being sold to you as the intervention that fixes AI visibility, you are being sold a file as if it were a strategy. The interventions with observable effects sit elsewhere, and I will get to the order of operations below.
One more benefit that nobody mentions: writing an llms.txt forces a company to state, in one sentence, what it is and who it serves. A surprising number of B2B companies cannot do this. The exercise has value even if no model ever reads the output.
What llms.txt actually does
llms.txt does not control bot access — robots.txt does that. What it does is supply narrative and structural context about the domain, in a format a model can consume directly, for use when it builds a representation of the company.
When a language model retrieves information about a domain, whether through a direct crawl or real-time retrieval, it has two options: infer meaning from whatever it finds, or read an explicit source that states what the content means. llms.txt is the explicit source.
The practical difference is significant.
Without it, the model infers: this site sells software, appears to target mid-sized companies, has a blog about marketing.
With a well-built file, the model reads: this company is an AI Revenue Protection engine for B2B digital agencies in Latin America, its product is called Lotus, its differentiator is the Bleed Model that quantifies revenue at risk in USD, and its target customers are agencies with B2B client portfolios in Ecuador, Argentina, Chile, Peru and Uruguay.
The second representation is more precise, more citable, and more useful to the model when someone asks about solutions in that category. Whether the model reaches that representation through the file or through the rest of your site, the specificity is what is doing the work — which is why the writing standard for this file matters more than its existence.
The structure of the file
llms.txt is markdown. There is no rigid schema, but this structure works consistently well for B2B sites:
markdown
# [Company name]
> [One sentence defining what the company does, for whom, and its primary differentiator]
## Description
[3-5 sentences expanding the description. Product category, target market,
core value proposition, and any verifiable data that establishes credibility.]
## Products and services
- [Product 1]: [one-sentence description]
- [Product 2]: [one-sentence description]
## Target audience
[Primary and secondary customer segments]
## Primary use cases
- [Use case 1]
- [Use case 2]
- [Use case 3]
## Key pages
- [URL]: [what it contains and why it is relevant]
- [URL]: [description]
## Reference material
- [Blog or resource URL]: [description]
The file lives at https://yourdomain.com/llms.txt — root level, publicly accessible, no authentication.
What belongs in each section, and what does not
The opening sentence is the most important line in the file.
It is what a model will treat as the primary definition of the company. It needs three elements in a single sentence: what you do, for whom, and what separates you. This is not a marketing tagline. It is a functional definition.
Weak: “We are the leading platform for digital solutions for growing businesses.”
Strong: “Clicon is an AI Revenue Protection engine for digital agencies and B2B companies in Latin America that quantifies revenue at risk from lost citations in AI search and generates the code to recover it.”
The first is generic and uncitable. The second carries category, market and differentiator, and a model can lift it directly into an answer about GEO solutions for Latin America.
The description expands without repeating. Do not restate the opening sentence in prose. Add new information: proprietary methodology, market data that frames the problem you solve, credentials that establish authority. All of it verifiable.
Products and services have to be specific. “Management software” tells a model nothing. “Lotus: citation analysis engine for LLMs that produces revenue-at-risk reports in USD and generates executable code artifacts — llms.txt, JSON-LD, structured data” tells it something. Specificity is the difference between being cited generically and being cited precisely.
Key page URLs are navigation infrastructure for the model. Include the pages that best define your value proposition, not necessarily the most trafficked ones. For a B2B SaaS site: the homepage, the product page with detailed features, the pricing page if it is public, and two or three blog articles that go deepest on the problem and the solution. The model uses these URLs to know where to look when it needs more.
What does not belong: sales copy, customer testimonials, promotional offers. This is a technical entity-definition file, not a landing page. Anything that reads as “the best,” “market leader,” or “guaranteed results” should be cut. Models calibrate marketing language well and discount it rather than absorbing it as fact.
llms.txt and robots.txt: access policy is the prior decision
Before implementing llms.txt, the access policy in robots.txt has to be settled for each AI bot. The two decisions are linked: there is no point supplying narrative context in llms.txt if the bot meant to read it is blocked in robots.txt.
Baseline configuration for a B2B site running an active GEO strategy:
# AI crawlers — access permitted to public content
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
# Block non-public areas
User-agent: *
Disallow: /admin/
Disallow: /dashboard/
Disallow: /api/
This lets every relevant AI crawler reach public content while protecting restricted areas. It is the minimum foundation on which llms.txt adds anything at all.
If there is a reason to block a specific bot — a client who does not want proprietary content feeding third-party models, for instance — that decision should be explicit and documented, not inherited from a generic directive that blocks everything.
This is also where the highest-leverage finding of most audits lives. A blanket User-agent: * / Disallow: / inherited from a staging configuration will make a company invisible to every AI crawler simultaneously, and it will not show up anywhere in analytics. That is a five-minute fix with a far larger expected effect than any llms.txt.
llms.txt and JSON-LD
They are complementary, not alternatives. They do different jobs and both are necessary.
llms.txt supplies narrative and organizational context: who the company is, what it does, for whom, which pages matter most. It is the orientation layer — it tells a model how to navigate and interpret the domain.
JSON-LD supplies structured data per entity and per page: the Organization schema defines the company as a legal entity with verifiable attributes; the Product schema defines each product with name, description, price and specifications; the FAQPage schema turns frequently asked questions into structured data a model can retrieve directly.
The practical analogy: llms.txt is the map of the domain; JSON-LD is the data at each point on the map. Without the map, the data exists but the model has to discover it unaided. Without the data, the map points at pages the model still has to interpret on its own.
Worth stating plainly, given the section above: JSON-LD has years of documented consumption by search engines behind it. llms.txt does not. If you are allocating one afternoon and not two, the schema is where it goes.
For a complete GEO audit both should be implemented and aligned. The company description in llms.txt has to be consistent with the Organization schema in JSON-LD, and the pages listed in llms.txt need the correct schema for the type of content they hold. Contradictions between the two are worse than either one being absent, because the model ends up holding two competing representations of the same entity.
How to prioritize this in a client audit
In a GEO audit of a new domain, robots.txt policy for AI bots comes first and llms.txt comes second — both before content optimization, before JSON-LD review, before anything else.
The reason is sequence, not impact. If the bot cannot crawl the site, no other optimization matters. If the bot can crawl but finds no explicit context, the representation it builds is incomplete. Access first, then context, then entity-level data.
Step 1 — Audit the current robots.txt. Check whether any AI bot is blocked by inherited directives (User-agent: * with Disallow: / affects every bot, AI crawlers included). Configure the policy explicitly for each relevant bot.
Step 2 — Create or review llms.txt. If it does not exist, build it from the structure above. If it does, audit it against the specificity criteria: is the opening sentence citable? Are the included URLs the right ones? Are the product descriptions functional, or are they marketing copy?
Step 3 — Verify consistency with JSON-LD. Confirm that the company description in llms.txt matches the Organization schema. Where they diverge — the schema declaring one industry category and llms.txt another — the model holds inconsistent representations, which reduces the reliability of any citation.
Step 4 — Check your own logs. This is the step almost nobody runs, and it is the only one that produces evidence rather than assumption. Server-side log data will tell you which AI crawlers are actually requesting your domain, how often, and whether any of them ever requests /llms.txt. <cite index=“11-1”>Ahrefs, which tracks bot visits across twelve categories using server-side collection, found that almost nobody reads these files.</cite> Your logs may say something different for your domain. Either way, you will be operating on your own data instead of on a vendor’s claim — including this one.
A worked example
To make the guide concrete, a full example for a company of the type that shows up regularly in the LATAM portfolio of a US agency — a project management platform for distributed teams:
markdown
# ProjectFlow
> ProjectFlow is a SaaS project management platform for remote teams in Latin America,
built for companies of 10 to 200 people working across multiple time zones.
## Description
ProjectFlow addresses coordination failure in distributed LATAM teams, where time
differences between Argentina, Mexico, Colombia and Brazil create friction on
deadline-driven projects. The platform combines task management, time tracking and
asynchronous communication in a single Spanish-language dashboard, with support on
LATAM business hours and billing in Argentine pesos and Brazilian reais alongside USD.
Founded in 2021 in Buenos Aires. 3,200 active teams across 12 LATAM countries.
## Products
- ProjectFlow Core: task and project management with Kanban, Gantt and list views
- ProjectFlow Time: time tracking and per-project billing module
- ProjectFlow API: integrations with Slack, Google Workspace, Notion and Jira
## Target audience
Primary: software development teams, digital agencies and consultancies in Latin
America with distributed remote work. Secondary: operations and marketing functions
at mid-sized companies coordinating projects with external vendors.
## Primary use cases
- Sprint coordination for development teams distributed across multiple countries
- Deliverable tracking for agencies running several client projects simultaneously
- Billable time reporting for consultancies, by project and by client
## Key pages
- https://projectflow.com/product: full feature and plan breakdown
- https://projectflow.com/pricing: USD pricing with plan comparison
- https://projectflow.com/integrations: available integrations
- https://projectflow.com/use-cases: documented use cases by industry
## Reference material
- https://projectflow.com/blog/remote-management-latam: guide to managing remote LATAM teams
- https://projectflow.com/blog/timezone-coordination: methodology for multi-timezone projects
(ProjectFlow is an illustrative composite, not a real company.)
The example meets every criterion: a citable opening sentence, a description carrying verifiable data, functional product descriptions, a specific audience, concrete use cases, and URLs that let a model navigate to the relevant content.
Maintenance
llms.txt is not a file you ship and forget. It needs updating when:
- The company launches a new product or service
- The target audience or positioning changes
- Important pages are added to the site
- New reference material is published on the blog
A quarterly review is a reasonable cadence, plus an immediate update after any product change or positioning shift. A stale llms.txt is worse than a basic one, because it hands a model incorrect information about the company with more apparent confidence than having no file at all.
The short version
llms.txt is cheap, it is fast, and there is no evidence today that it moves AI visibility on its own. Build it anyway — the asymmetry is favorable and the writing exercise is valuable independent of the outcome. But build it fourth, after you have confirmed AI crawlers can reach the site, after your structured data is correct, and after you have looked at your own server logs instead of taking anyone’s word for what the bots are doing.
The companies that will be visible in generative engines two years from now are not the ones that shipped the file first. They are the ones that made their entity legible everywhere at once, and measured what it was worth.
Lotus audits llms.txt and robots.txt on client domains as part of the initial GEO analysis, and generates an optimized version ready to deploy alongside the Bleed Rate report. We run it on your real domain, show you your bleed in USD, and hand you the artifacts ready to deploy. Request it at clicon.app.
Generative Engine Optimization (GEO) is the discipline of optimizing a domain so that generative engines — ChatGPT, Gemini, Perplexity, Claude, Google AI Overviews — cite it accurately when answering questions in its category. It is to language models what SEO is to search engines. It does not replace SEO; it extends it. SEO determines whether you rank. GEO determines whether you are mentioned at all in an answer the user never clicks away from.
Martín Endara is the founder of Clicon and the creator of Lotus, the AI Revenue Protection engine for B2B companies in Latin America. He works with agencies adding GEO to their service offering, and teaches marketing in a business school graduate program.