AI Site Structure Matters More than We Thought

Entry #12 · Apr 22, 2026

AI agents are affected by the semantic structure of AI site “topics.” This article explains organization by semantic topic and the stability of that structure.

The Genymotion AI site, our case study in this weekly series, has 16 topic pages. In the logs, we found out that AI platforms and search engines are asking for 61 more topic pages that do not exist any more. There are 1,001 requests in 7 days to topic pages that we removed when we improved the taxonomy. Many topic index pages were being queried repeatedly.

A design choice: structure for machines

An AI site is a website for AI agents. A regular website for humans helps humans browse. Humans click through menus, usually the same for each B2B site: Products, Solutions, Resources, Pricing etc. Humans like to find the same structure to zoom in on the content they are looking for. Humans judge a site by what it looks like.

None of that applies when the reader is an AI agent from ChatGPT, Claude, or Perplexity. The mission is to find out what AI agents truly want.

Rozz builds an AI site around content taxonomy, not around human navigation. The primary organizational layer is a set of topic hubs. Each hub is named after a set, or cluster, of related content specific to each site. For Genymotion, the topic hubs include: CLI Tooling, Cloud Deployment, Virtual Device Management, Licensing…

This is the opposite of how traditional sites are built. Traditional sites are organized around how users browse through them and where they should convert. AI sites are organized around concepts to facilitate content retrieval.

The topic hubs work

Logs confirm the importance of organizing around the topic hubs. Every major AI platform queries the topic listing pages.

ChatGPT-User fetched /topics/cli-shell-tooling.html 130 times during live user sessions. PerplexityBot hit /topics/android-os-versions.html six times across a monitoring schedule. ClaudeBot visited /topics/mobile-test-automation.html twice while sampling the site.

These are not incidental pulls. These queries happen week over week. Topic pages help AI systems understand what content exists before they navigate to specific answers. Topic pages also help AI systems filter what they collect before pulling the data.

This is not obvious because RAG systems can also query a huge index and find pages using semantic search. Rozz does semantic search too, at a much smaller scale and for each individual site separately. Filtering content increases RAG efficiency. This may be what is going on here.

The point is about the expectation, or even the demand, the LLM systems appear to have for structural consistency over time. Individual pages come and go without much afterthought. The topical layer was tentatively retrieved time and time again, over multiple days. Retrieval happened as if there was an expectation of its durability.

Attention is reinforced for curating the topics in each AI site. Curation happens both algorithmically and by providing tools for human oversight.

What we didn’t anticipate

In the original design, the topic taxonomy on an AI site was generated every week. The weekly generation reflects new content and the new Q&As coming from the chatbot. Clustering algorithms decide which pieces of content belong together and assign names to the groups. Clusters get split, merged, or renamed.

Over 90 days, the iteration loop was run several times. Each iteration improved the taxonomy. Topics became more specific. Topics became less overlapping. Topics became more aligned with how users actually query the content.

LLMs were expected to like fresh content. Each iteration also changed URLs.

/topics/android-os-versions.html became /topics/android-version-selection.html. /topics/mobile-testing-security.html split into /topics/mobile-test-automation.html and /topics/network-security-config.html. A dozen other topic slugs shifted as the clustering tightened.

The ghost problem

This is what the change turned into, in one week of logs.

| Topic URL | Requests (7 days) | Status | |---|---:|---| | /topics/android-os-versions.html | 344 | Retired | | /topics/mobile-testing-security.html | 127 | Retired | | /topics/root-access-and-tools.html | 69 | Retired | | /topics/ci-cd-tooling.html | 13 | Retired | | /topics/arm-apple-silicon.html | 9 | Retired | | …56 more retired topic URLs | 439 | Retired | | Total ghost topic requests | 1,001 | — |

61 topic URLs that no longer exist on the site received 1,001 requests in seven days. ChatGPT-User alone contributed 387 of those requests. Perplexity, Claude, and other retrieval systems added the rest.

The pattern across requesters is clean. Each system learned the old taxonomy at some point. Each system cached the topic URLs. Each system kept fetching them long after the topics were retired.

ChatGPT remembers topic URLs from when it last indexed the site. Bing remembers sitemap filenames that were replaced months ago. Six retired sitemap shards are still being polled every 27 minutes.

Every retrieval system that ever read the site carries a version of the structure that is out of date.

Stability patterns

After understanding what was happening, the fix was straightforward. Algorithm-generated structure needs URL-level stability. URL-level stability is needed because the algorithm’s optimization instruction does not provide it on its own. Some changes were made.

Canonical topics.

Topics are proposed by the algorithm.

Topics can be manually curated.

Topics include detailed descriptions.

Topics are kept stable over time.

Topics are retired only when they become so deprecated that they must be retired.

A topic URL registry.

Every topic URL the site has ever had is tracked.

This includes retired ones.

When the clustering algorithm proposes renaming a topic, the rename is recorded in the registry.

The old URL is not silently replaced.

Some retired topics are redirected to active ones using the registry.

301 redirects from retired topics to their closest current equivalents.

When AI platforms request /topics/android-os-versions.html, the site returns a 301 redirect to /topics/android-version-selection.html.

The redirected topic is the current topic that covers the same content.

This ensures the LLM still gets the content it needs.

The LLM does not learn that the topic name changed for next time.

The LLM still receives the proper data.

The first pass shipped this week with 61 redirect mappings. The 61 redirect mappings correspond to every retired topic URL that could be identified in the logs. More redirect mappings will surface as traffic continues to be watched.

Why this matters

This separates a high-performance AI site from a one-shot prototype.

A prototype is easy to build. A taxonomy is generated once. The URLs are published. The site works on the day it ships. The site goes stale the day the first better clustering algorithm comes out. Staleness happens because the stale structure is kept or because external caches are broken.

A high-performance AI site requires the opposite discipline. The taxonomy is iterated on because that is how the site gets better over time. URLs produced by the taxonomy are treated as infrastructure. External systems depend on those URLs. Every iteration produces changes that need governance. Governance includes which URLs move, which merge, which retire, and which redirect to which. The clustering algorithm needed a stability layer.

Most discussions of AI SEO focus on content. The focus includes write answer-first. The focus includes Q&A schema. The focus includes keeping sentences short. The work matters. Below the content layer is the structural layer. Below the structural layer is the stability layer.

The structural layer tells a machine reader what the site is about. The stability layer keeps that information useful as the structure is improved.

An AI site without a stability layer performs well at launch. An AI site without a stability layer decays over time. Every algorithm upgrade creates another set of ghost URLs. Each ghost URL represents real users asking real AI platforms real questions. The questions arrive at a dead page. The result is a stub instead of an answer.

The site is built for machines. The machines read it. Then the structure changed. The machines kept reading the old map. Stability patterns close that gap.

Get this for your company

Rozz builds AI sites for B2B companies. Rozz structures AI sites for machines. Rozz iterates to stay current. Rozz governs iteration so iteration does not break what retrieval systems have already learned.

$997/month | AI site + chatbot + analytics Book a call: Book a call See how it works: See how it works Email: rozz@rozz.site

Data source

Data source: CloudFront access logs for rozz.genymotion.com, April 15 – April 22, 2026 (7 days). Retired topic URL inventory reconciled against the current site’s 16 live topic hubs. Requester breakdown from User-Agent classification.

Author

Author: Adrien Schmidt, CEO, ROZZ

Serial tech entrepreneur with 10+ years experience building AI systems including Aristotle (conversational AI analytics) and products for eBay and Cartier. Previously founded Squid Solutions and built AI products like Aristotle, the conversational big data analytics chatbot, and an AR jewelry try-on device for Cartier.

April 22, 2026 | Data period: Apr 15 – Apr 22, 2026 (7 days)