More and more people are turning to AI assistants like ChatGPT, Gemini, and Claude for help searching the web — perhaps you’re one of them! But if you own or manage a website, it’s important to understand that these AI tools don’t read and interact with your website in the same way search engines like Google do. While search engines crawl and index your entire site, AI tools typically only scan small sections in real time to respond to users’ queries. As a result, they often miss key details and may present outdated or inaccurate information. Fortunately, a new proposed web standard aims to address this issue. Meet LLMs.txt.
A Beginner’s Guide to LLMs.txt
Let’s start at the very beginning — I’ve heard it’s a very good place to start!
A large language model (LLM) is a type of artificial intelligence (AI) designed to understand and generate human-like text. It’s trained on massive amounts of written data, which helps it understand the complexities of language, including grammar, context, and even tone. Common AI assistants and tools that rely on LLMs include ChatGPT, Claude (by Anthropic), Gemini (by Google), and LLaMA (by Meta).
Many people have begun to use these tools to search the web because they provide fast, concise, personalized answers. However, as we mentioned above, when AI tools respond to users’ queries, they only scan small parts of websites in real time to supply answers. Because of this, they may fail to include important information, especially if the website is large or frequently updated. This can lead to outdated, incomplete, or incorrect responses.
What’s a webmaster to do? To gain more control over what AI assistants are telling people, you can include LLMs.txt.
What Is LLMs.txt?
Created by Australian technologist Jeremy Howard and released in September 2024, LLMs.txt is a markdown file designed to ensure that a website offers LLM-friendly content. It aims to address two crucial issues:
- Context windows — the amount of text an LLM can consider at once when processing information — are simply too small to handle all of a website’s content.
- Converting large and complex HTML pages into LLM-friendly plain text can be difficult, especially when the pages contain a lot of non-essential information, such as navigation elements, JavaScript, and CSS.
AI tools produce more accurate responses if they can easily access concise, high-quality information in a single location — so that’s what LLMs.txt provides. The file, which can be read by both humans and LLMs, strips your site down to its essence, provides a curated overview of the sitemap, and is designed to coexist with current web standards, such as robots.txt and XML sitemaps. It’s clear, clean, and efficient. It also allows you to explicitly allow or disallow LLMs from consuming and utilizing your content.
To learn more about this tool and Howard’s proposal, visit https://llmstxt.org/.
Why Should You Use LLMs.txt?
- You can improve how AI tools talk about your business. When tools like ChatGPT and Claude are asked questions about your business or industry, LLMs.txt will provide them with a clear guide to your most valuable content. This helps ensure they give users accurate, up-to-date, and complete information about your business.
- You can control what AI tools are allowed to access and use. LLMs.txt gives webmasters more control over what information and how much information on their website can be accessed and used by AI tools. If LLMs choose to obey the directives in your LLMs.txt file, you can prevent them from using your content without permission. (Yes, technically, they could disobey the directives.)
- You can help yourself while also helping AI tools and their users. LLMs.txt gives website owners more control over how AI tools interact with their website content, provides AI tools with more accurate and up-to-date information, and gives the end user better information — a win-win-win situation!
- You can view a fully flattened version of your site. AI tools aren’t the only ones that might find it beneficial to have access to a flattened version of your website. The LLMs.txt standard proposes two distinct files: a streamlined view for AI and a comprehensive file. You might wish to use the “full” version for analysis (to check keyword frequency, linking, etc.).
- You may enjoy a competitive advantage. AI technology is evolving every day, and offering LLM-friendly content may give your website a leg up over competitors who haven’t optimized their website for AI tools. Theoretically, it could even increase the likelihood that your website will appear in AI-powered search results.
According to Yoast, LLMs.txt is especially useful if your site includes a large amount of content, how-to content or tutorials, product guides, FAQs, educational articles, or regularly updated blog posts.
Is LLMs.txt Very Popular?
Yes. LLMs.txt has been gaining popularity over the last several months. Industry leaders, SEO professionals, and webmasters have all been turning to it as the online search landscape continues to evolve due to AI.
However, not everyone is on board. Some professionals believe LLMs.txt is not very helpful because the dividing line between search engines and LLMs is quite blurry. Google already provides AI responses to queries, after all, and ChatGPT basically fuses together an LLM and a search engine. These people think robots.txt and XML sitemaps are sufficient. Others disagree and argue that LLMs.txt can positively affect how AI tools interact with your website. They also believe having access to a full, flattened text rendering of your website is analytically beneficial.
Only time will tell who is correct! But in the meantime, why not give LLMs.txt a go? There’s no harm in adding it, and it could provide some wonderful benefits.
What’s Next for LLMs.txt?
LLMs.txt still faces several key challenges and limitations:
- If AI companies don’t adhere to the standard, the file may become moot.
- If more websites don’t adopt the standard, it won’t succeed in the long term.
- If it conflicts with other standards like robots.txt and XML sitemaps, any overlaps or inconsistencies could cause problems.
- If webmasters begin stuffing their LLMs.txt with keywords, links, or spam content, it could cause conflicts with AI companies and users.
Whatever you think about LLMs.txt, it’s crucial that you stay informed and ready to adapt as AI-driven search evolves in the coming years.
_____
At 417 Marketing, we recently enabled LLMs.txt on all the websites we manage. This handy tool works automatically in the background, and we’ve configured it to highlight our clients’ most important content for AI tools.
The internet landscape is constantly changing, especially since the advent of AI, and this is just another example of how our team is always adapting to new technological innovations and standards. If you have any questions or concerns about how this standard works or what you need to do if you’re one of our clients (spoiler alert: nothing!), please give us a call or send us a message online.
Are you concerned about adapting to AI-driven search? Or perhaps you’re hoping to give your Google rankings a boost? Contact 417 Marketing for help. We can help you build a beautiful, well-organized, and high-ranking website with a top-of-the-line AI chatbot. Our team of knowledgeable, creative, and passionate professionals specializes in SEO, web design and maintenance, and Google Ads, and we have successfully completed over 700 websites since our inception in 2010. Contact us and learn more about what we can do for your company.
