To create an llms.txt file that passes the PageSpeed Insights Agentic Browsing check, place a plain-text markdown file at the root of your domain (e.g., https://example.com/llms.txt) containing structured context, primary page mappings, and interaction directions. This standard allows AI agents to rapidly understand your site structure, target actions, and API capabilities without scraping heavy layout files.
What is an llms.txt File?
The llms.txt file is an emerging, community-led standard for web-scale machine readability. It serves as a manifest for large language models (LLMs) and autonomous agents, similar to how robots.txt governs classic search crawlers. When an agent crawls your site, it checks for llms.txt to receive a lightweight, markdown-formatted directory of your site’s main informational nodes and transactional endpoints.
PageSpeed Insights audits for llms.txt in the Agentic Browsing category because it decreases the token consumption and parsing time required for an agent to successfully evaluate a domain.
Step-by-Step Implementation Guide
Follow these steps to deploy a compliant llms.txt file on your server:
- Structure the File Layout: Divide it into Title and Description, Root Context, and Secondary Pages & Actions.
- Create a Detailed Sub-file (llms-minor.txt): For large sites with extensive developer docs, keep the main llms.txt file small (under 100 links) and link to a secondary manifest called llms-minor.txt for deeper directory trees.
YouTube Shorts Script Blueprint
Repurposing Template (50-Second Script)
- [0:00 – 0:10] Hook: “There is a new hidden file you need to add to your website root right now to help AI search bots read your pages.”
- [0:10 – 0:25] The Problem: “It’s called llms.txt. PageSpeed Insights is now auditing websites for it under the ‘Agentic Browsing’ category. Without it, AI agents consume too many tokens scraping your site and may skip you.”
- [0:25 – 0:40] The Fix: “It’s easy: create a plain-text markdown file named llms.txt. Describe your business in one sentence, list your 5 most important page links with descriptions, and upload it to your root directory.”
- [0:40 – 0:50] Call to Action: “Need an llms.txt template? Visit Lexington Digital and read our full step-by-step developer tutorial.”