The AI-native files at the center of today’s GEO advice drew under 1% of crawler traffic. The bots that answer users touched them least of all. The main audience turned out to be scrapers and tooling.
What we measured
Open any current guide to GEO / AEO and you will find the same checklist item.
Publish an llms.txt, add structured JSON feeds, and hand the machines a clean, machine-readable map of your site.
The premise is intuitive.
Models are machines.
Machine-readable files are easy for machines to parse.
Machines should prefer them.
We had a rare chance to test that premise against real traffic. The premise does not hold.
Across a set of production AI-native sites we operate and instrument, we measured bot access for a 90-day window. We measured 461,328 requests across 31,935 log files. We decoded each user-agent user-agent by user-agent. We sorted each user-agent into more than twenty classes. The AI crawlers included training, indexing, and live citation fetchers from OpenAI, Anthropic, Perplexity, Meta, and others. The classes also included the traditional search engines. The classes also included generic scrapers. The classes also included human or tooling traffic.
These sites are AI-native in that they serve exactly what the industry best practices recommend.
The sites serve an llms.txt, an llms-full.txt, and JSON feeds for search, pages, topics, and Q&A.
We wanted to measure what is useful and what is not.
To be clear, plenty of literature exists that calls the llms.txt files into question.
Plenty recommends them.
In fact, the whole GEO/AEO space is fluid.
Everyone wants to do it.
No one knows how to do it for sure.
So we measured on our own domains.
Finding one: for everyone, it is a rounding error
Of all 461,328 requests, just under 1% touched the AI-native files at all.
llms.txt drew 0.30%.
The JSON feeds drew 0.67%.
The combined share was 0.97%.
The other ninety-nine percent went to ordinary HTML pages and sitemaps.
That pattern holds across every crawler.
Here is the share of each one’s own traffic that went to llms.txt or the JSON feeds, highest first:
| Crawler | Category | AI-native reqs | Total reqs | Share | |---|---|---:|---:|---:| | OAI-SearchBot | AI indexing | 75 | 3,092 | 2.43% | | GPTBot | AI training | 140 | 7,131 | 1.96% | | browser / tooling | human | 2,604 | 215,079 | 1.21% | | CCBot | AI training | 24 | 2,321 | 1.03% | | Baiduspider | search engine | 56 | 7,223 | 0.78% | | Googlebot | search engine | 26 | 4,140 | 0.63% | | YandexBot | search engine | 8 | 1,500 | 0.53% | | PerplexityBot | AI indexing | 6 | 1,149 | 0.52% | | Applebot | search engine | 31 | 6,678 | 0.46% | | generic scraper | scraper | 380 | 88,946 | 0.43% | | Meta | AI training | 53 | 13,366 | 0.40% | | Bingbot | search engine | 57 | 15,024 | 0.38% | | Amazonbot | AI training | 72 | 22,774 | 0.32% | | DuckDuckBot | search engine | 7 | 2,206 | 0.32% | | curl | scraper | 40 | 20,978 | 0.19% | | python | scraper | 9 | 4,846 | 0.19% | | ChatGPT-User | AI citation | 21 | 15,092 | 0.14% | | http-lib | scraper | 14 | 12,171 | 0.12% | | ByteSpider | AI training | 6 | 5,930 | 0.10% | | ClaudeBot | AI training | 2 | 11,070 | 0.02% |
The single most engaged crawler anywhere is OpenAI’s SearchBot. OpenAI’s SearchBot spends 2.4% of its requests on the structured files. OpenAI’s GPTBot is next at 2.0%. After that, the share falls off a cliff. Every other bot and every search engine and every scraper sits under 1.2%. Most shares are under half a percent. There is no crawler in the data for which the AI-native layer is anything but a trace.
So the foundational premise of these files is not borne out. The premise is that bots will favor structured feeds over HTML. No single crawler we logged follows that premise. All crawlers run on HTML.
Finding two: the index crawlers actually use is the sitemap
Here is the part that reframes the exercise.
The case for llms.txt is that machines need a clean, machine-readable index of a site instead of parsing cluttered HTML page by page.
That argument is half right.
Crawlers do lean on a machine-readable index.
It is not llms.txt.
It is the sitemap, a standard that has existed since 2005.
Break all 461,328 requests down by what was fetched:
| Path | Requests | Share | |---|---:|---:| | HTML (pages, Q&A, homepage) | 254,752 | 55.2% | | sitemap.xml | 94,338 | 20.4% | | robots.txt | 11,082 | 2.4% | | JSON feeds (/api/*.json) | 3,094 | 0.67% | | llms.txt | 1,403 | 0.30% | | other (redirects, assets, scanner noise) | 96,659 | 21.0% |
HTML, sitemap, and robots together are 78% of everything bots fetch.
The two AI-native inventions, llms.txt and the JSON feeds, are 0.97% combined.
The sitemap alone outdraws llms.txt by roughly sixty-seven to one.
So llms.txt fails because it is a second index for a job the sitemap already does.
Finding three: the bots that answer users ignore it hardest
If the structured files were going to matter anywhere, the structured files would matter at the moment of citation. The citation moment is when a model fetches a page to answer a live user. We see the opposite.
The closer a bot sits to producing a user-facing answer, the less the bot touches the AI-native layer.
ChatGPT-User, OpenAI’s live answer-time fetcher, made 15,092 requests in the window. Twenty-one requests were for the AI-native files. That was 0.14%.
Anthropic’s Claude-User made one fetch.
Anthropic’s ClaudeBot made more than 11,000 requests.
Anthropic’s ClaudeBot fetched llms.txt zero times.
Anthropic’s ClaudeBot hit the JSON feeds twice.
That rate was 0.02%.
These are the bots whose output users actually read. These bots consume your HTML. These bots effectively skip the layer built specifically for them. These bots also skip the sitemap too. ChatGPT-User’s sitemap rate is 0.0%.
The live fetcher does not browse an index at all. The live fetcher does not browse a machine-readable index. The live fetcher does not browse any index. The live fetcher goes straight to the single page the model wants.
Indexes are for the crawlers that build a map in advance. Indexes are not for the agent answering in the moment. This means even the sitemap is an indexing lever. The sitemap is not a citation one.
What this does and does not show
The scope matters because this is easy to over-read.
The data shows narrow and solid results.
The machine-format add-on files, llms.txt and JSON feeds, earn negligible engagement from every category of crawler.
The data also shows near-zero engagement from the ones that generate answers.
If a plan is to bolt these files onto a site and expect the citation pipeline to consume them, the pipeline is not consuming them.
The data does not show what HTML quality is like. The data does not show what content structure is like. The data does not show whether a dedicated AI site beats an ordinary one. Every site serves HTML. Bots reading HTML is not an argument for any particular kind of site. These questions are separate. These questions belong to a separate piece.
One fair caveat for skeptics is measurement scope.
The measurement is fetch volume, not influence.
In principle a model could read llms.txt once.
The model could let llms.txt shape a whole answer.
Low volume is not the same as zero value.
The volume for answer-producing bots is low. The volume is 0.14% and below. Even a generous influence-per-fetch assumption leaves the layer touching a vanishing slice of what those bots do. The measurement cannot see inside the models. The measurement can only see what bots request. The bots request HTML.
Caching is not hiding the number either.
These logs are CDN edge logs.
CDN edge logs record cache hits and origin fetches.
A low llms.txt count is a real low count.
A low llms.txt count is not traffic quietly absorbed before it reaches the log.
One scoping note in the other direction is the measurement target.
These are requests to the AI-native sites we serve.
If a site also published llms.txt from its own origin, fetches from that origin would not appear here.
This is a large, representative sample of how bots treat the files. This sample is not a census of the entire web.
A note on our own position follows.
We build and serve these files on the sites we instrument.
This instrumentation is exactly why we can measure the files at this resolution.
This finding partly critiques our own product surface.
We would rather publish the measurement than the marketing.
Whatever the value of an AI-native site turns out to be, it is not the llms.txt.
The takeaway
The AI-native add-on layer does not pay off. Across 461,328 requests and more than twenty crawler types, the files at the center of the standard GEO checklist drew under 1% of traffic. The files were ignored hardest by the bots that answer users. The files found their main audience among scrapers and tooling. The machines that were built for these files are reading pages instead.
There is a constructive reading underneath the negative one.
Crawlers are not allergic to structure.
Crawlers consume a machine-readable index of a site all day.
It is the sitemap.
The sitemap gets sixty-seven times the traffic of llms.txt.
So the effort that goes into llms.txt and JSON feeds is better spent on the two things every crawler actually ingests.
The first ingested thing is the HTML pages themselves.
The second ingested thing is a clean, complete sitemap.
An AI site does not just produce AI-discovery files. This is a small part of it. An AI site produces optimally formatted, rewritten HTML pages. The HTML pages are organized in a new sitemap. The sitemap includes semantic groupings. An AI site also includes lots of actual user-generated Q&A. In the logs, this is what is actually fetched.
Get this for your company
Rozz gives visibility into the AI conversations happening about your product. Rozz also gives the tools to influence what AI recommends.
$997 /month | AI site + chatbot + analytics
→ Book a call | → See how it works | → rozz@rozz.site
Data source: CloudFront edge access logs across the production AI-native sites ROZZ operates, a 90-day window. 461,328 requests over 31,935 log files, classified by decoded user-agent into 20+ crawler classes and by fetched path. Fetch volume only (cache hits and origin fetches). No claim about per-fetch influence.
Serial tech entrepreneur with 10+ years experience building AI systems including Aristotle (conversational AI analytics) and products for eBay and Cartier. Previously founded Squid Solutions and built AI products like Aristotle, the conversational big data analytics chatbot, and an AR jewelry try-on device for Cartier.
June 29, 2026 | Data period: 90-day window, 461,328 requests
rozz @ rozz.site | © 2026 ROZZ. All rights reserved.