Technical foundations
Should your website add an llms.txt file?
Decide when llms.txt is worth maintaining, build a small pilot, and distinguish file requests from search visibility.
TL;DR
- Treat llms.txt as an optional guide to useful content, with a named owner and a clear use case.
- Fix inaccessible pages, missing facts and broken links before investing in this extra file.
- Test whether your intended tools can use it; a file request does not prove a citation or a sale.
An llms.txt file can help an agent find relevant documentation. Add it when you know which content it should point to. Assign someone to keep that content current and test a real use case. It is a weak priority when the main product pages still lack clear answers.
The llms.txt proposal describes a Markdown guide with context and links. Its August 2026 revision permits both a root file and scoped files such as /docs/llms.txt. It remains a proposal, rather than a universal search requirement.
What does the evidence justify?
In an Ahrefs study of 137,210 domains with traffic in May 2026, 28% published a valid llms.txt file. Among files that existed, 97% received no requests that month. The sample consisted of Ahrefs Web Analytics users, so it does not represent every website. Ahrefs methodology and results.
The useful distinction is between publishing a file and someone using it. Among requests that did reach llms.txt files, 19.5% came from named AI tools. Those fetches did not establish that the contents influenced an answer. These observations support a small, measured experiment rather than a visibility promise. Ahrefs request classification.
Google separately says new AI text files are not needed for its AI search features. That is a Google-specific requirement statement, not a test of every agent. Google AI feature guidance.
Use this decision sheet before implementation
The following is an editorial planning tool. Use the criteria to decide where to spend effort. They are not ranking factors.
| Your situation | Recommended decision | Evidence to keep |
|---|---|---|
| Public API documentation with agent users | Pilot a scoped documentation index. | A real task, current links and the test output. |
| Marketing pages missing product facts | Repair the pages first. | Approved specifications and readable page text. |
| A maintained file already exists | Check its accuracy and usage before expanding it. | Broken-link report and request logs. |
| No team can own updates | Defer a separate index. | The content owner and update process still needed. |
Worked example: a small documentation pilot
Illustrative example: a fictional sensor vendor with public developer documentation.
The support team keeps getting the same integration question: which authentication guide applies to its current API? The vendor has a current guide, an archived guide, and a useful error reference. Its navigation gives both API versions similar prominence.
First, the documentation owner confirms the current version and repairs the navigation labels. The developer then creates /docs/llms.txt with this short map:
# Northline Sensor API
> Public documentation for the current Northline Sensor API.
Use the current API guide for new integrations. Archived guides describe older releases.
## Current integration guides
- [Authentication](https://example.com/docs/authentication.md): Set up credentials for the current API.
- [Error reference](https://example.com/docs/errors.md): Interpret errors returned by the current API.
## Optional
- [Archived API](https://example.com/docs/archive.md): Historical reference only.
The example follows the proposal's basic format. The vendor must publish and maintain the linked pages before it shares the index. Changing a file extension does not create a Markdown page.
To accept the work, the owner opens every link, compares the version labels with the source docs, and confirms the file returns its intended text. A tester then gives the index to the intended agent and asks the authentication question. The test checks whether the agent reaches the right guide and gives a supported answer. It does not test automatic discovery by every platform.
Track file requests, correct test answers, search citations and site visits separately. Revisit the pilot when the API changes or the file stops matching the docs.
llms.txt: common questions
Does publishing the file control training permission?
No. The file describes content. Crawler preferences belong in the relevant platform controls and robots.txt rules. See the crawler policy approach.
Should we publish the entire website as one file?
Start with a small index and useful destinations. A large combined document adds another copy to maintain and makes it harder to identify which page is authoritative.
How long should implementation take?
Scope the work from the state of the content. A small index may be simple; repairing inaccurate documentation and maintaining parallel formats can take much more work.
Semanta can help decide whether an index solves a real discovery problem or merely adds maintenance. Bring your key pages, documentation workflow and one agent task to discuss the right technical priorities.