LLM Wikis Are Over-Engineered

LLM Wikis Are Over-Engineered — I Replaced Mine With a Pure Python Compiler

Large language models (LLMs) are powerful tools for generating text, answering questions, and assisting with coding. But over the past year, a curious trend emerged: developers started building "LLM wikis" — massive, complex knowledge bases designed to feed context to models. These wikis often involve vector databases, embedding pipelines, retrieval-augmented generation (RAG) stacks, and Kubernetes clusters. The irony? Most of this complexity is unnecessary for personal or small-team use.

I fell into the trap myself. After spending weeks configuring ChromaDB, deploying a FastAPI server, and wiring up LangChain, I realized my setup was brittle, slow, and overkill for what I actually needed: a simple way to store and retrieve structured facts for an LLM to use during inference. So I scrapped it all and built a pure Python compiler that reads flat Markdown files and outputs a lightweight, queryable knowledge base. No databases, no embeddings, no Docker. Just Python, a folder of notes, and a few hundred lines of code.

Here’s why LLM wikis are over-engineered, and how you can replace yours with a simpler, faster, and more maintainable solution.

The Problem With LLM Wikis

LLM wikis promise to solve a real problem: LLMs have limited context windows and no built-in memory for your specific data. So you build a system that pre-processes your documents, stores them as vectors, and retrieves the most relevant chunks when you ask a question. This approach works, but it introduces unnecessary complexity for most use cases.

Why Most Teams Don’t Need Vector Databases

Vector databases like Pinecone, Weaviate, or Qdrant are designed for large-scale similarity search across millions of documents. If you’re running a customer-support chatbot for a global enterprise, sure, you need that. But for a personal wiki with a few hundred notes, a vector database is like using a freight train to move a backpack. The overhead of setting up embeddings, managing indexes, and tuning chunk sizes often outweighs the benefits.

The Hidden Costs of RAG Pipelines

Retrieval-augmented generation adds latency, cost, and failure points. Every query requires: 1. Embedding your question 2. Searching the vector index 3. Retrieving top-k chunks 4. Passing them to the LLM with your prompt

If any step fails (network timeout, embedding model error), your LLM gets no context. And if your chunks are poorly chosen, the LLM might ignore them entirely. I’ve seen RAG pipelines where the retrieval step alone took 2–3 seconds — unacceptable for interactive use.

When Simplicity Wins

For most individuals, small teams, or internal tools, the actual requirement is simpler: give the LLM a curated set of facts relevant to the current task. You don’t need semantic search when you already know what you want to ask. You need a way to organize your knowledge so the LLM can access it without a complex retrieval system.

A Simpler Approach: Pure Python Compilation

Instead of building a retrieval system, I built a compiler. The idea is straightforward: write your knowledge as structured Markdown files in a folder. The compiler reads all files, extracts sections and metadata, and generates a single compressed text file that you can feed directly into an LLM prompt. No databases, no embedding models, no external services.

How It Works

The compiler does four things: 1. Scans a directory for `.md` files 2. Parses each file into sections (using headings as keys) 3. Combines everything into a flat, compressed text format 4. Outputs a single file that fits within your LLM’s context window

You can then prepend this compiled knowledge to your prompts. For example:

You are a helpful assistant. Here is the relevant knowledge base:
--- BEGIN KNOWLEDGE ---
[compiled output here]
--- END KNOWLEDGE ---
Answer the user's question based only on this knowledge.

Why This Works

This approach works because most personal wikis are small enough to fit in a single LLM context window. GPT-4 Turbo supports 128k tokens — that’s enough for a book-length knowledge base. Even with smaller models, you can compress aggressively: remove formatting, shorten variable names, and use abbreviations. The trade-off is that you lose semantic search, but for structured facts (API docs, configuration notes, project guidelines), exact heading-based retrieval is often sufficient.

Requirements

Before we dive into the code, here’s what you need:

• **Python 3.10+** (for structural pattern matching and type hints)
• **A folder of Markdown files** — your existing notes, docs, or wiki
• **No external dependencies** — we use only the standard library

That’s it. No Docker, no databases, no cloud services.

Step-by-Step Installation

1. Create the project structure

First, create a folder for your compiler:

mkdir ~/llm-compiler
cd ~/llm-compiler
mkdir knowledge

Place your Markdown files inside the `knowledge` directory. For example, create a file called `api-reference.md`:

# API Reference

## Authentication
Use Bearer token in Authorization header.

## Endpoints
### GET /users
Returns list of users. Requires admin role.

### POST /users
Creates a new user. Body: { "name": string, "email": string }

2. Write the compiler script

Create a file named `compile_knowledge.py`:

#!/usr/bin/env python3
"""Compile a folder of Markdown files into a single compressed knowledge base."""

import argparse
import os
import re
from pathlib import Path
from typing import Dict, List

def parse_markdown_file(filepath: Path) -> Dict[str, str]:
    """Parse a Markdown file into sections keyed by heading."""
    with open(filepath, 'r', encoding='utf-8') as f:
        content = f.read()
    
    sections: Dict[str, str] = {}
    current_heading = "GENERAL"
    current_lines: List[str] = []
    
    for line in content.splitlines():
        heading_match = re.match(r'^(#{1,6})\s+(.+)', line)
        if heading_match:
            if current_lines:
                sections[current_heading] = '\n'.join(current_lines).strip()
            current_heading = heading_match.group(2).strip()
            current_lines = []
        else:
            # Skip empty lines at start of section
            if line.strip() or current_lines:
                current_lines.append(line)
    
    if current_lines:
        sections[current_heading] = '\n'.join(current_lines).strip()
    
    return sections

def compile_knowledge(knowledge_dir: Path) -> str:
    """Compile all Markdown files into a single compressed string."""
    compiled_parts: List[str] = []
    
    for md_file in sorted(knowledge_dir.glob("*.md")):
        sections = parse_markdown_file(md_file)
        filename = md_file.stem
        
        # Add file header
        compiled_parts.append(f"## FILE: {filename}")
        
        for heading, body in sections.items():
            # Compress: remove excess whitespace, shorten headings
            compressed_body = re.sub(r'\s+', ' ', body).strip()
            short_heading = heading.replace(' ', '_').lower()[:40]
            compiled_parts.append(f"[{short_heading}] {compressed_body}")
    
    return '\n'.join(compiled_parts)

def main():
    parser = argparse.ArgumentParser(
        description="Compile Markdown knowledge base for LLM prompts"
    )
    parser.add_argument(
        "--input", "-i",
        default="./knowledge",
        help="Directory containing Markdown files (default: ./knowledge)"
    )
    parser.add_argument(
        "--output", "-o",
        default=None,
        help="Output file path (default: print to stdout)"
    )
    args = parser.parse_args()
    
    knowledge_dir = Path(args.input)
    if not knowledge_dir.exists():
        print(f"Error: Directory {knowledge_dir} does not exist")
        return 1
    
    compiled = compile_knowledge(knowledge_dir)
    
    if args.output:
        with open(args.output, 'w', encoding='utf-8') as f:
            f.write(compiled)
        print(f"Compiled knowledge written to {args.output}")
    else:
        print(compiled)
    
    return 0

if __name__ == "__main__":
    exit(main())

3. Make it executable

chmod +x compile_knowledge.py

Usage Examples

Basic compilation

Run the compiler on your knowledge folder:

python compile_knowledge.py --input ./knowledge

Output:

## FILE: api-reference
[authentication] Use Bearer token in Authorization header.
[endpoints] ### GET /users Returns list of users. Requires admin role. ### POST /users Creates a new user. Body: { "name": string, "email": string }

Save to a file for reuse

python compile_knowledge.py --input ./knowledge --output compiled_kb.txt

Use the compiled knowledge in a prompt

Now you can prepend this compiled text to any LLM prompt. Here’s a Python example using the OpenAI API:

import openai

with open("compiled_kb.txt", "r") as f:
    knowledge_base = f.read()

system_prompt = f"""You are a helpful assistant with access to a knowledge base.
Use the following knowledge to answer questions. If you cannot find the answer, say so.

--- BEGIN KNOWLEDGE ---
{knowledge_base}
--- END KNOWLEDGE ---"""

response = openai.ChatCompletion.create(
    model="gpt-4-turbo",
    messages=[
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": "How do I authenticate to the API?"}
    ]
)

print(response.choices[0].message.content)

Expected output:

To authenticate to the API, use a Bearer token in the Authorization header.

Advanced: Auto-compile before each session

Add this to your shell configuration (`.bashrc` or `.zshrc`) to auto-compile your knowledge base before starting a new LLM session:

alias compile-kb='python ~/llm-compiler/compile_knowledge.py --input ~/llm-compiler/knowledge --output ~/llm-compiler/compiled_kb.txt'

Then run `compile-kb` before opening your LLM interface.

When This Approach Breaks Down

No solution is perfect. This pure Python compiler has limitations:

• **Large knowledge bases** — If you have tens of thousands of pages, you’ll exceed context limits. In that case, you need a retrieval system.
• **Dynamic knowledge** — If your knowledge changes frequently, you must recompile. For static documentation, this is fine.
• **Semantic search** — You can’t ask fuzzy questions like “what was that thing about rate limiting?” You must know the section heading or browse the compiled output.

For these cases, consider hybrid approaches: use this compiler for your core, stable knowledge, and add a simple keyword search (using Python’s `difflib` or `re`) for fallback retrieval.

Conclusion

LLM wikis are over-engineered for most personal and small-team use cases. The complexity of vector databases, embedding pipelines, and RAG stacks often adds more problems than it solves. By building a pure Python compiler that transforms your Markdown notes into a compressed, prompt-ready format, you gain:

• **Simplicity** — No dependencies beyond Python’s standard library
• **Speed** — Compilation takes milliseconds, not seconds
• **Control** — You know exactly what goes into your prompts
• **Maintainability** — Your knowledge is just Markdown files, easy to edit with any text editor

This approach won’t replace enterprise-scale knowledge management systems. But for individuals, small teams, and anyone tired of debugging Docker containers for a personal wiki, it’s a liberating alternative. Start with a folder of Markdown files. Add a compiler. Skip the database. Your LLM won’t know the difference — and you’ll sleep better at night.

*For further reading on LLM best practices and simple architectures, see the OpenAI News blog, Google AI Blog, and Microsoft AI Blog. The broader trend toward minimalism in AI tooling is well-documented in these sources.*