What AI bots read, what they ignore, and what an AI site is actually for. Part 1: Citation bots

Source: https://rozz.site/insights/what-bots-read-and-ignore.html

Published:

Keywords: AI bots, citation bots, AI site, ChatGPT-User, Claude-User

Document ID: rozz_site_insights_what-bots-read-and-ignore.html

Entry #14 · May 19, 2026

What AI bots read, what they ignore, and what an AI site is actually for. Part 1: Citation bots

We built rozz.genymotion.com to market to AI agents, like genymotion.com markets to humans.

Let’s take a look at what the bots actually fetch and what they don’t.

Then we’ll pull some learnings out of all this.

Teaser: Q&As are pretty popular (61% corpus coverage in 21 days). llms.txt is not (0 ClaudeBot fetches; sitemap.xml got 209).

What’s an AI site?

rozz.genymotion.com exists alongside genymotion.com.

The marketing website serves humans.

The marketing website is organized in a standard way.

The marketing website includes Products, Resources, Pricing, and related sections.

The AI site serves AI agents. The AI site serves ChatGPT, Claude, Perplexity, and the crawlers that feed them. The AI site presents content in a different way. The different way is a research direction being explored in this Insights series.

We’ve been writing about weekly findings after analyzing the AI-site logs in this blog series. This article zooms out. After four months of building and iterating, we use the past three weeks of clean bot-log data. The bot-log data spans Apr 29 – May 19, 2026. The bot-log data covers ChatGPT-User and Claude-User. The bot-log data includes only 200-OK responses. The analysis tries to make sense of what an AI site actually is.

Quick scope note: we have bot logs for rozz.genymotion.com. Everything below is about what bots do on the AI site. We are not making claims about the human site.

What we built

The rozz.genymotion.com AI site structure includes:

Each Q&A page corresponds to a question a user actually asked.

The runbooks include gmsaas for cloud and gmtool for desktop.

Some content is for the AI agent making a query on behalf of a human user. Some content is for a coding agent itself. Runbooks exist for a coding agent. Genymotion is a tool for developers. Some content is for crawler bots that build the search indexes those AI agents draw from.

The point of an AI site is that multiple consumers exist. The AI site caters to each consumer. The AI site explores what sticks.

In this article, we focus on the citation bots. Citation bots provide the most direct business value. Citation bots occur during actual conversations with users.

What citation bots fetched

In the 21-day window, ChatGPT-User and Claude-User did 1,517 content fetches between them.

| Slice | Share of fetches | Share of corpus | |---|---:|---:| | Top 1 URI | ~8% | 0.1% | | Top 10 | ~37% | 1.2% | | Top 50 | ~78% | 6.2% | | Top 100 | ~93% | 12.4% |

A small number of URIs do most of the work.

What sits in the head

The most-fetched single content URI is the pricing Q&A.

The pricing Q&A has 123 fetches.

After the pricing Q&A, the next most-fetched items include: the rooted-device Q&A, the system-requirements Q&As, the SaaS-vs-Desktop comparison, and the macOS compatibility Q&A.

On the pages side, the Burp Suite security-testing tutorial has 130 fetches. The topic listing /topics/android-version-selection has 251 fetches as a topic listing.

Together, these items represent what the head of the AI site does. The head focuses on pricing, requirements, compatibility, and a security-testing tutorial.

It seems like the pro-enterprise skew introduced in article 13 is paying off.

Two surfaces, two patterns

We currently have 2 main types of content pages.

One type is Q&A (/qna).

The other type is cleaned website pages.

If Q&As are separated from pages, two different shapes appear.

| | /qna/ | /pages/ | |---|---:|---:| | Corpus | 262 | 545 | | Fetched URIs | 160 | 132 | | Coverage | 61% | 24% | | Total fetches | 930 | 587 | | Top 1 share | 13% | 22% | | Top 10 share | 41% | 53% | | 1-fetch share (of fetched) | 35% | 51% | | Dark inventory (never fetched) | 39% | 76% |

Q&As get fetched 2.5× more proportionally than pages. The Q&A distribution shows more items being fetched.

The pages Pareto distribution is much steeper. The Burp Suite tutorial accounts for 22% of all pages fetches. Three-quarters of the page corpus is unread in three weeks.

Q&As are answers. Pages are sources.

Q&As are the answers

Each Q&A in the corpus was generated from a real chatbot conversation.

Every Q&A title corresponds to a question someone actually asked.

The title corresponds to the person’s actual words.

The idea is to conform the Q&A to actual user queries in the AI engines.

Top 10 Q&As, last 21 days

| Fetches | Q&A |

|---:|---|

| 123 | what-pricing-plans-are-available-for-genymotion |

| 66 | does-genymotion-provide-rooted-device... |

| 50 | how-much-memory-do-i-need-to-have-20-virtual-devices |

| 26 | what-are-genymotion-desktop-s-system-requirements |

| 24 | what-are-the-system-requirements |

| 23 | what-are-genymotion-s-pricing-options-for-saas-and-desktop |

| 19 | what-are-the-costs-for-using-genymotion-saas |

| 19 | does-the-emulator-work-with-the-latest-mac-os |

| 18 | how-do-i-install-genymotion-desktop-on-windows-macos-or-linux |

| 17 | how-to-run-the-emulator-in-the-cloud |

These Q&As reflect what people ask AI about Genymotion. The questions occur when people are considering whether to use Genymotion.

A real tail exists underneath the head. 100 Q&As were fetched once or twice each in the window. Examples of single-fetch Q&As include:

Each example corresponds to one specific person evaluating Genymotion. Each evaluation corresponds to one specific commercial use case. The Q&A got fetched once by one AI session. The fetch happened on behalf of one buyer.

This looks like a long-tail content pattern. The top 10 Q&As account for 41% of fetches. The remaining 150 fetched Q&As account for the other 59%.

The tail is bigger than the head.

Pages are the sources

The /pages/ corpus behaves differently.

The Burp Suite tutorial is by far the most accessed page.

The Burp Suite tutorial has 130 fetches.

The Burp Suite tutorial is 22% of all pages fetches.

After the Burp Suite tutorial, the numbers drop fast. The Genymotion documentation hub has 48 fetches. The Linux install guide has 27 fetches. A handful of requirements, install, and root-access pages fall in the 12–17 fetch range.

122 pages got 10 or fewer fetches. More than half of those pages got exactly one fetch each. 413 pages (76%) got nothing.

Does the result mean the entire /pages corpus is unnecessary? The role of a corpus with mostly non-directly-fetched items remains unclear in the question.

We think the answer is in one stat. 79% of sessions that fetched a Q&A also fetched at least one of the source pages cited in that Q&A’s “Based on these sources” sidebar. This indicates that the model verifies the Q&A answer. The verification happens by pulling the cited source page in the same session. The /pages give proper credibility to the Q&A pages.

This also means the silent 76% is not dead weight. Each page is a potential source citation for a Q&A. When a Q&A gets asked, the page gets fetched. When a Q&A is not asked, the page sits.

Looking ahead, we could trim pages that are not mentioned in the sources sidebar, if any.

Page single-fetch tail samples

A simple way to understand the /pages content is the supply of product and marketing information available to the Q&A. Don’t cut it. Let the market decide which ones to keep.

Summary

Q&As are demand-driven.

Q&As are fetched as primary answers.

Q&As are distributed in a real long tail.

Pages are supply-driven. Pages are fetched as source citations. Pages are distributed with a concentrated head and a large dormant body.

They play different roles. They are complementary.

What citation bots didn’t fetch

Citation bots rarely fetch the discovery layer.

The discovery layer includes reference files such as robots.txt, sitemaps, llms.txt, and various JSON APIs that were published.

Citation bots do not browse around the website. Citation bots fetch URLs that the model already knows.

The discovery work is done by crawler bots upstream. The crawler bots include ClaudeBot, GPTBot, OAI-SearchBot, and PerplexityBot. Those crawler bots build indexes. The citation bots draw from those indexes.

This is what the data shows. 39% of ChatGPT-User sessions start with a direct content-URL fetch. The bot knew the URL it wanted before the session began.

> A teaser for the next articles: those AI bot crawlers seem to heavily prefer traditional SEO infrastructure over the AI-native discovery standard. In the same 21-day window, ClaudeBot fetched sitemap.xml 209 times and llms.txt zero times. OAI-SearchBot and GPTBot fetched sitemap.xml 16 times combined and llms.txt twice each. We published the AI-native files because the standard exists but our observations show that they’re not really used yet.

What the fetched content looks like

In our 21-day sample, some consistent patterns emerged.

Here are some insights.

Decomposition by product line

The popular Q&As split the answer by product line.

The product line includes Genymotion Desktop: and Genymotion SaaS (Cloud):.

The popular Q&As also split the answer by persona. The persona includes Ideal for: Occasional users, pilots. The persona also includes Ideal for: Enterprises needing bespoke setups.

We assume the LLM can easily pick the content it needs to answer the question.

Tables for comparisons

The SaaS-vs-Desktop Q&A is an 8-row × 3-column table.

The table covers Hosting, Scalability, Collaboration, Automation, Maintenance, Use Cases, Cost Model, and Security & Compliance.

The rooted-device Q&A uses a 3×3 task / need / how table.

It seems like LLMs are good at reading tables.

User-voice question titles

Some Q&A titles preserve typos.

Some titles preserve missing words.

Some titles preserve run-ons.

Some titles preserve broken grammar.

The titles are made of real chatbot input. That input is real.

The wording of the pages has been rewritten according to fluency rules. The rewritten form uses bullets-not-paragraphs. The rewritten form uses tables-not-prose. The rewritten form uses every-sentence-standalone. The rewritten form avoids anaphora such as “this”. The rewritten form avoids anaphora such as “the above”. The rewritten form avoids cross-paragraph dependencies.

The compressed form is less pleasant to read for humans. That compressed form is not the point.

The target reader is an AI agent. The AI agent extracts content. The AI agent makes its own prose when chatting with a human.

The human-readable prose still exists. It lives on genymotion.com.

The AI site is generated from that material. The generation uses the rewriting pipeline.

Canonical URLs on every AI-site page point back to genymotion.com. The AI site is not a replacement for the human site. The AI site uses the same content for a different kind of user.

What this means for a B2B company building an AI site

Three takeaways hold up against the data.

1. Q&As do more work than pages, and the gap is large. Q&As have 2.5× higher coverage. Q&As have gentler Pareto. Q&As have a real long tail.

The Q&As generated from observed user questions earn more bot traffic per page than rewriting existing content. The Q&As represent user demand. User demand is what users are actually asking about. The pages are used for verification. Both are needed. The contributors to citations differ.

2. The compressed form is the content; the marketing prose lives elsewhere. An AI site is not a human website with structured data added. An AI site is a parallel surface. The content is rewritten for extraction.

The human site keeps doing the human job. The AI site is a different deliverable. The AI site has a different consumer.

3. Don’t bet on AI-native discovery as your primary channel yet. Publish llms.txt. Expose JSON APIs.

Fine. The work in mid-2026 is still being done by robots.txt and sitemap.xml. The work is similar to the last 25 years of web SEO. Use traditional SEO infrastructure to reference the AI site.

Bots still use it. Bots are still using traditional SEO infrastructure.

The simpler version of all three is: an AI site is a separate property with a different design target than a marketing website. The content shape differs. The navigation differs. The discovery layer differs. The audience differs.

Treating an AI site as “the same content with JSON-LD” misses what it actually is. It also misses what bots actually use.

Get this for your company

Rozz gives you visibility into the AI conversations happening about your product.

Rozz also provides tools to influence what AI recommends.

$997/month | AI site + chatbot + analytics

→ Book a call → See how it works → rozz@rozz.site

Data source: CloudFront access logs for rozz.genymotion.com, April 29 – May 19, 2026 (21 days). ChatGPT-User and Claude-User content fetches only. Responses are 200-OK. Corpus inventory reconciled against the live AI-site URI registry.

Author

Serial tech entrepreneur with 10+ years experience building AI systems including Aristotle (conversational AI analytics) and products for eBay and Cartier.

Previously founded Squid Solutions and built AI products like Aristotle, the conversational big data analytics chatbot, and an AR jewelry try-on device for Cartier.

May 19, 2026

Data period: Apr 29 – May 19, 2026 (21 days)

rozz @ rozz.site | © 2026 ROZZ. All rights reserved.