Map

get

Truth

Vectors

find

Scraps

A RAG Sitemap is a structured, AI-readable knowledge index designed for AI retrieval. It is not sitemap.xml and not a vector database. It is a traceable context, a top-level navigation document that marks the boundaries and relationships between content groups and describes the whole knowledge landscape of a site, so AI walks the structure toward the answer instead of gambling on semantic similarity.

A humanoid robot in a scholar's robe points to a line in an open book, showing it to a human author at the desk who holds a quill, and the author leans in to look closely at that spot, with drafts and notes spread across the desk. The reversal of the two figures' roles is the key to this painting: the robot is not a student being examined but a copy editor giving the site a check-up. Its finger points to an unclear line of description, and that spot is the gap itself. The quill stays in the human's hand; the right to revise has not been handed over, and the AI only lets the person see where the writing is not clear enough. The book pulled half-out on the desk hints that this is not a large-scale rewrite but a point-by-point fine-tuning of titles, category descriptions, and article placement, all the moves a site owner makes in ordinary SEO.

Use an AI Chatbot to Train Your Site for SEO

You think you are training the AI to understand your site? It should be the other way around: use an AI chatbot to train your site. Every time the AI answers wrong, it is telling you which article's title or description is unclear, helping you check your site's SEO blind spots. RAG Sitemap drops the black-box vector store and reads the titles, categories, and descriptions you already wrote in WordPress, generating a plain-text site map for the AI. You only fix things in the admin the way you already do SEO, with no new tool to learn and no algorithm to read minds.

A Renaissance horseman hauls hard on the reins at the edge of a cliff, the horse's front hooves skidding to a halt a moment before the fall, solid ground behind and a deep dark gorge ahead. The horse represents the LLM's natural generative force, a high-entropy gallop in itself. The reins are the harness, a set of deliberately applied engineering constraints. The decisive instant happens at the entrance of each hop: if any one Sub Agent gives way, the whole chain of reasoning plunges into the gorge of accumulated error. The rider does not suppress the horse but takes the direction back into his own hands, which is also the role of the Diving Agent, holding the Master Sitemap to decide the dive point for the whole team. The solid ground behind corresponds to the static cache segment, where the prompt, Master Sitemap, and chunks stand unmoving.

RAG Harness Engineering

RAG Harness Engineering means every visitor question triggers not a single AI prompt, but three independent AI API calls: vision, retrieval, answer. Chaining multiple sub-agents normally risks each stage poisoning the next, but the Harness architecture hands every stage the visitor's original question and a clear view of the initial task goal, making it fundamentally immune to contamination. Accumulated noise is stopped at the entrance of every hop.

一個渺小、穿著樸素希臘長袍的人形機器人，手提一盞小燈，在一座龐大宏偉的石柱長廊中自信地往深處走去，廊柱朝遠方無止盡延伸。渺小的機器人是 Llama 3B 這種垃圾級小模型，手中的小燈只照亮自己腳下，是它有限的世界知識。但牡步伐自信，因為真正在引路的是周圍的石柱秩序，不是手裡的燈。柱列朝深處延伸，對應 master → category → post 的漸進式披露。模型小不要緊，秩序夠清楚的時候，每一 hop 都收斂成一道選擇題。這幅畫的主角不是機器人，是廊柱本身，能力強弱不是關鍵，結構正確才是。

A Small Language Model (SLM) Actually Understood an Entire Website

The small model Llama 3.2 3B is a language model with only 3B parameters, about as small as they come. Ask it a question and it can only answer from its 3B of training data. It does not know what your site says, does not know which articles you published, and knows nothing about the content you have built up recently. Using it to run website Q&A should have been a fantasy.

A small figure holds only a single flat paper map, trapped inside a huge multi-level labyrinth of stone arches and staircases, endless stairs extending up, down, and in every direction. The flat paper map is the essence of llms.txt: a catalog with only titles and no body, meant for flat reading. The three-dimensional maze is the knowledge structure of a real site, with floors, circulation routes, and interconnected depth. The figure is tiny, and the mismatch in scale between tool and environment is exactly the situation of an AI holding a 2D list against 3D content. Light enters from one side, but without the right map, light cannot replace structure. What RAG Sitemap sets out to solve is making this flat plan three-dimensional, walking down the three layers of master, category, and post.

The Good-Faith Limits of llms.txt

llms.txt is a sitemap designed for AI to read, but its limit is that it has only one layer, which is not enough for an organized, structured site. Its standard format is the site name as an H1, a short summary, and below that each line is a title: description pointing to one page. But it cannot tell whether a line is a category page, a standalone page, an article, or a product page; every line is treated as the same kind of thing. The intention is not wrong, and the goal is to make a site easier for AI to read. The problem is not the description but that it flattens the site into a single layer, destroying the site's original narrative power and content context.

Progressive Disclosure

An index built for on-site conversation that doubles as how AI search engines understand you.
The chatbot follows it to find answers; the AI search engine follows it to find you.

Progressive Disclosure

An index built for on-site conversation that doubles as how AI search engines understand you. The chatbot follows it to find answers; the AI search engine follows it to find you.

The logic of building the whole picture from a structured summary first, then pinpointing and retrieving in depth along the clues in the question, already appears in many places. Google never reads the entire web before ranking. It looks at the title and H1 first, then the meta description, then Schema.org structured data, and only then goes into the body text. This is the inevitable result of cost control. The LLMs.txt idea that began circulating in 2024 identified the problem correctly, but most current implementations are just a list of links. An AI handed that list swallows the whole thing at once and tastes nothing, no matter how good the content is.

The Skill.md format Anthropic introduced for Claude Code improves the same thing: a short description at the top of a project file lets Claude load the project's full context only when needed, instead of loading all of its knowledge every time. The core assumption behind GEO (Generative Engine Optimization) is the same. When an AI search engine indexes a site, it uses a lightweight model to scan quickly and decide which content is worth going into, and the input to that filtering layer is your site map. These ideas point to one conclusion: in an age of information overload, what AI needs most is not more data but a carefully designed progressive-disclosure architecture.

The core design principle of a RAG Sitemap is called Progressive Disclosure. Rather than feeding all content to the AI at once, it guides the AI layer by layer along the site's own category structure, from the whole down to the detail. The AI first reads the Master Sitemap, the top-level index, to understand the site's knowledge structure: which content groups exist, what topic each group covers, and where the boundaries between groups lie. It then selects the matching content group based on the type of question, retrieves precisely within that group, and finally reads only the truly relevant content fragments to generate an accurate answer.

— RAG Sitemap is the complete realization of this idea at the site level.

Progressive Disclosure: An Indexing Philosophy That Has Never Changed

JSON-LD is read by search engines, Claude's SKILL.md is read by AI agents, and a RAG Sitemap is read by the on-site AI chatbot. The reader changes all the way from crawler to LLM, yet all three systems face the same challenge of quickly deciding what is most relevant to the current need, and all three converge on the same three-layer structure: title, description, content.

Perspective One · The Three Fields Inside a Single Unit
Lightweight scan Precise deep dive
Google Search
Web-wide crawling and ranking
1
Title
Schema.org name
2
Description
Schema.org description
3
Content
Schema.org articleBody
Claude SKILL.md
Agent task context
1
Title
YAML name
2
Description
YAML description
3
Content
SKILL.md
RAG Sitemap
On-site knowledge retrieval
1
Title
Entry Title
2
Description
Entry Description
3
Content
the entry's corresponding chunk
{
  "@context": "https://schema.org",
  "@type": "Article",

  "name": "iPhone Review",

  "url": "example.com/iphone",

  "description":
    "In-depth review...",

  "articleBody":
    "Overall performance..."
}
---
name: pdf-processing

description: Extract text and
  tables from PDF files
---

# PDF Processing

Call process_pdf(filepath)
to start processing...
======
Title: iPhone Review

Link: example.com/iphone

Description: Complete review of
  iPhone's core features
======

Content:
  Overall performance,
  good battery life...
Any retrieval system facing the constraint of limited resources and a need to quickly judge what is worth reading in depth arrives at the same answer. The title is for identification, the description is for judging relevance, and the content is the thing actually read. The RAG Sitemap Generator is itself a set of entropy-reduction tools that can easily convert a WordPress site's existing category structure into a plain-text site map an AI can read.
Perspective Two · The Three-Layer Structure of the File System

The three-layer structure of Perspective One holds inside a single unit of content. Pull the lens back to the whole file system and the same three layers appear again: index, category, file. Progressive disclosure does not happen only inside an article; it also decides how the entire site is organized.

Top level Leaf nodes
XML Sitemap
Site directory structure
1
Index
sitemap.xml
2
Category
/products/, /blog/ and other subdirectories
3
File
Individual HTML pages
Claude SKILL.md
Skill folder
1
Index
SKILL.md
2
Category
scripts/
3
File
scripts/*.py executables
RAG Sitemap
RAG directory structure
1
Index
master-sitemap.txt
2
Category
category-list/
3
File
post_*.txt、page_*.txt
example.com/
├── sitemap.xml     # index
├── products/       # category
│   ├── cat-a/
│   │   └── item.html
│   └── cat-b/
└── blog/           # category
    └── post.html
pdf/
├── SKILL.md        # index
├── FORMS.md
├── reference.md
├── examples.md
└── scripts/        # category
    ├── analyze.py
    ├── fill.py
    └── validate.py
rag-sitemap/default/
├── master-sitemap.txt # index
├── category-list/  # category
│   ├── cat_x.txt
│   └── cat_y.txt
├── post-chunks/    # category
│   ├── post_x.txt
│   └── post_y.txt
└── page-chunks/    # category
    └── page_x.txt
This structure recurs in the file system because it is not a design choice but the natural solution humans reach for when organizing large amounts of information: a book has a table of contents, chapters, and sections; a library has a master catalog, classes, and books; a website has a home page, categories, and articles. RAG Sitemap did not invent this file hierarchy. It simply converts the three layers any well-structured site already has into the format best suited for an AI to read.