Lettuce Docs Tags Graph

The LLM-wiki pattern

#docs#concepts
~8 min read

The LLM-wiki pattern

An LLM-wiki (a pattern popularised by Andrej Karpathy) is a knowledge base that an LLM builds and maintains on your behalf, rather than one you fill in by hand. You feed it raw material; the agent compiles that material into a cross-linked set of pages, keeps a catalog and a ledger, and answers questions against what it has compiled — filing the good answers back so the wiki gets smarter over time.

Lettuce is built to be a substrate for exactly this. The wiki is just a folder of Markdown notes, and every operation an agent needs is exposed as an MCP tool. This page explains the three layers, then maps the INGEST / QUERY / LINT operations onto Lettuce’s tools.

The three layers

A well-run LLM-wiki is really three layers stacked on top of each other:

1. sources/ — the immutable raw layer

Everything that came from outside the wiki lands in sources/: ingested articles, pasted transcripts, downloaded papers. These files are treated as append-only and never rewritten — and this is enforced, not just convention: through the agent API an agent may create a new source or append to one, but overwriting, patching, deleting, or moving an existing file under an immutable prefix is rejected. The default prefix is sources/; a vault can change it with an immutable: list in index.md frontmatter (or opt out with immutable: []), and get_vault_info reports the active prefixes. They’re the evidence the wiki is built from, so you can always trace a claim back to where it came from. Structural files like sources/** are exempt from the content lint rules — they aren’t expected to be tidy, cross-linked notes.

2. The wiki — the LLM-owned compiled layer

This is the wiki proper: the entity, concept, and summary pages the agent writes by reading the sources and synthesising them. A page about a person, a project, a technical concept, or a recurring question. These pages are dense, cross-linked with [[wikilinks]], and meant to be re-read and revised. This is the layer the agent owns and continuously improves.

3. The schema — CLAUDE.md / AGENTS.md

The rules of the wiki — its conventions, its tag taxonomy, how pages should be named and structured — live in CLAUDE.md (the maintainer guide that lettuce wiki init scaffolds) and/or an AGENTS.md brief. Lettuce surfaces these to every connected agent through the get_vault_info tool, which returns the two files concatenated as a single brief. So the first thing an agent learns when it connects is how this particular wiki wants to be maintained.

Note

CLAUDE.md and AGENTS.md play the same “schema” role — the names just match whichever agent tool you use. get_vault_info surfaces both. Only the root brief is surfaced; there is no per-folder AGENTS.md.

Two structural files tie the layers together:

Scaffolding a wiki

lettuce wiki init [dir] [--force] [--agents]

lettuce wiki init scaffolds exactly four files into the directory:

With --agents it also writes a tailored AGENTS.md (vault-etiquette brief generated from whatever the directory currently contains). --force overwrites files that already exist. It does not write any .lettuce/*.yml config.

Tip

wiki only takes init. The actual work commands — ingest, search, lint, slides, agents — are top-level lettuce subcommands, not subcommands of wiki.

The operations

INGEST — turn raw material into compiled pages

Ingest is a two-step move: get the raw source in, then compile it.

  1. Capture the source. lettuce ingest <url | file | dir> takes a URL, a local file, or a directory (markdown, text, HTML, PDF, docx, odt, rtf, epub, org, rst, or latex), converts it to clean Markdown — readability for web pages, PDFKit text extraction for PDFs, pandoc for the other formats — downloads the article’s images, writes sources/<slug>.md, and appends a dated entry to log.md if one exists.

  2. Compile + cross-link. The agent reads the new source and writes the durable pages. For an entirely new page, file_page is the workhorse: in a single call it writes the page, links it under index.md’s ## Pages heading (configurable via index_section), and appends a dated log.md entry — so the catalog and ledger stay correct automatically. To extend an existing page, the agent uses patch_section (append / prepend / replace under a heading or block) or write_note in append mode, keeping the cross-references right.

So file_page is what makes the wiki compound: every new page is born already cataloged and logged.

When the agent runs under a propose-only key (its writes queue for review instead of committing — see Suggestions mode/01 Overview), a multi-page ingest can be filed as one reviewable unit with propose_batch: the user accepts or rejects all the new pages together instead of one at a time.

QUERY — answer questions, and bank the good answers

Querying is search → read → synthesise, with one extra step that makes the wiki grow:

  1. search runs full-text search (FTS5/BM25) over note titles and bodies and returns {path, title, snippet, score} — score is the BM25 rank, where lower is better. From the shell the same index is available as lettuce search <vault> <query…> [--limit N], which supports "quoted phrases", uppercase OR / NOT, and a leading - to exclude a term.

    • semantic_search complements it with vector recall — find pages by meaning, not just keyword: it embeds the query and ranks by cosine similarity, with a hybrid mode that fuses the two via reciprocal-rank fusion. Build the vector index once with lettuce embed <vault> (any OpenAI-compatible embeddings endpoint, including a local one like Ollama — no key, nothing leaves the machine). Use keyword search for exact names, semantic for “pages about X”.

    • answer closes the loop in one call: it retrieves the most relevant material and asks the model for a grounded, cited answer (from the shell, lettuce answer <vault> <question…>). With embeddings configured it retrieves at passage granularity — ranking individual section-sized chunks and feeding the model just the matching passages (grouped back by source page for citation) rather than whole notes, so the context is tighter and more on-topic. Build the chunk index with lettuce embed.

  2. read_note (and read_outline / link_graph to gather neighbours) pulls the matching pages.

  3. The agent synthesises an answer.

  4. File the answer back. When an answer is durable — a question worth keeping the answer to — the agent files it as its own page with file_page. The next query finds it directly instead of re-deriving it. This is the compounding loop: good answers become pages, pages become future search hits.

LINT — keep the wiki healthy

Two kinds of lint, one deterministic and one LLM-assisted:

A companion read tool, suggest_tags, proposes tags for the untagged / missing-required notes that lint flags (also via lettuce lint --suggest). Like lint_semantic, it’s propose-then-confirm and needs a provider.

Putting it together

A day in the life of an LLM-wiki:

  1. lettuce ingest https://… drops a cleaned article into sources/ and logs it.

  2. The agent reads it, then calls file_page to write a concept page — cataloged in index.md, logged in log.md, cross-linked to related pages.

  3. Later you ask a question. The agent searches, read_notes the hits, answers — and file_pages the answer so it’s there next time.

  4. Periodically, lint catches broken links and orphans; lint_semantic flags a contradiction between two pages, which the agent reconciles and records in log.md.

The result is a vault that gets denser and more useful the more you feed it — and stays inspectable, because everything is plain Markdown, the catalog is one file, and the ledger is greppable.

See also