teamai-test/.teamai/skills/common/kb-retriever/README.md

Kb Retriever Skill — Local Knowledge-Base Retriever

A skill for AI agents to efficiently answer questions over a local, multi-format knowledge directory using hierarchical index navigation and progressive retrieval — without ever loading whole files into context.

中文文档 · Back to collection root

What it does

Point the agent at a local directory full of mixed-format files (Markdown, PDF, Excel, …) and ask questions in natural language. The skill:

1. Walks a hierarchical index of data_structure.md files to figure out which files are likely to contain the answer.
2. Forces a learn-before-process step when it hits a PDF or Excel — it must read the corresponding references/*.md first and use the recommended tool, instead of blindly reading the whole file.
3. Retrieves progressively with grep + small windowed reads (offset/limit) instead of dumping entire files into context.
4. Iterates up to 5 rounds, narrowing keywords each round until it has enough evidence to answer.

---

Core features

• ✅ Multi-format: Markdown / text, PDF, Excel — extensible per file type.
• ✅ Hierarchical index: each directory carries its own data_structure.md, forming an index tree the agent navigates.
• ✅ Progressive retrieval: grep-first, windowed reads, never whole-file loads — keeps token usage low even on large corpora.
• ✅ Mandatory learning step: PDF/Excel processing is gated on reading the right references/*.md first.
• ✅ Bounded iteration: at most 5 retrieval rounds, with explicit termination conditions.

---

Skill structure

skills/kb-retriever/
├── SKILL.md                            Main skill (frontmatter name: kb-retriever)
├── README.md  /  README.zh-CN.md       This document
├── references/
│   ├── pdf_reading.md                  How to handle PDFs (pdftotext / pdfplumber / pypdf)
│   ├── excel_reading.md                How to read Excel with pandas (nrows, dtype, etc.)
│   └── excel_analysis.md               How to filter / aggregate / derive metrics on Excel
└── scripts/
    └── convert_pdf_to_images.py        Convert PDF pages to images when text extraction fails

---

Setting up your knowledge base

This skill does not ship a knowledge base — you bring your own. Two ways to wire it up:

Default location

Put a knowledge/ directory at the root of the workspace where you invoke the agent:

your-project/
├── .claude/skills/  or  .agents/skills/
│   └── kb-retriever/             ← this skill folder
└── knowledge/                    ← ← ← your knowledge base
    ├── data_structure.md         (root-level index, see template below)
    ├── <domain-1>/
    │   ├── data_structure.md
    │   └── ...
    └── <domain-2>/
        └── ...

Custom location

Tell the agent which path to use in your question, e.g. "answer from ./docs" or "my knowledge base is at /data/kb". The skill will use that path instead.

If the default knowledge/ does not exist and the user hasn't specified a path, the skill will ask rather than guess.

data_structure.md template

Each indexed directory should carry one of these:

# [Directory name]

## Purpose
What this directory is for and when it should be searched.

## Files
- file1.pdf — what it contains, time / version range
- file2.xlsx — schema summary, key columns
- subdir/ — what lives in this subdirectory

## Coverage
Time range, version, source, anything else that helps the agent prioritize.

---

How it retrieves

1. Hierarchical index navigation

For each directory level the skill reads data_structure.md, picks the most relevant child(ren) for the user's question, and recurses — so it doesn't fan out across the whole tree.

2. Learn before process (PDF / Excel)

When the candidate set contains a PDF or Excel file, the skill must first read the corresponding reference doc:

✅ Read references/pdf_reading.md  /  excel_reading.md  /  excel_analysis.md
✅ Understand the recommended tool & flags
✅ Convert / extract the file with that tool
⏭️  Then start retrieving

Forbidden:

• ❌ Trying to process a PDF without reading pdf_reading.md
• ❌ Trying to process an Excel without reading excel_reading.md / excel_analysis.md
• ❌ Skipping the conversion step and grepping the raw binary

3. Progressive retrieval

• Don't read whole files.
• Use grep to locate keywords first.
• Read only the matching window (limit ≈ 200–500 lines).
• Iterate up to 5 rounds, refining keywords.

4. Per-format tool strategy

Format	Tool	Notes
Markdown / text	grep + windowed read_file	Always offset/limit; never whole-file.
PDF	pdftotext input.pdf output.txt → grep on the text	Always extract to a file, never to stdout. Use -f / -l for page ranges on huge PDFs.
Excel	pandas with nrows first to learn schema, then filtered reads	Identify key columns (id / time / category) before querying.

5. Iteration loop

Each round:

1. Generate / update keywords
2. Pick under-explored candidate files
3. Run grep / windowed reads
4. Inspect snippets
5. Decide: enough to answer? → stop. Otherwise iterate.

Stops on either: answer found ✅, or 5 rounds reached ⏱️.

---

Best practices

Recommended

1. Always start from data_structure.md.
2. Read the matching references/*.md before touching a PDF or Excel.
3. Retrieve from the most relevant file first; expand only if needed.
4. Use offset + limit to read precise windows.
5. Extract PDFs to files, then grep — never paste the binary into context.

Avoid

1. ❌ Reading entire large files in one go.
2. ❌ Processing PDF / Excel without reading the references first.
3. ❌ pdftotext input.pdf - (stdout) — eats tokens.
4. ❌ Loading a whole Excel sheet at once.
5. ❌ Blind search across all directories.

---

FAQ

Q1: Why force the agent to read references/*.md first? To make sure the agent uses the right tool with the right flags — otherwise it tends to either dump huge files into context or pick a slow / broken tool.

Q2: How do I handle a very large PDF? Use page-ranged extraction (pdftotext -f 1 -l 10), grep the resulting text, then read only the matching pages.

Q3: Can my knowledge base live anywhere? Yes. Just say so in your question: "answer from /data/my-kb".

Q4: How do I improve retrieval accuracy? Use specific keywords, narrow down with time / file-name hints, and prefer domain-specific terminology over generic words.

---

Tool requirements

The skill assumes the agent has access to:

• grep — text search
• read_file — windowed reads with offset / limit
• pdftotext (poppler) or pdfplumber — PDF text extraction
• pandas — Excel reads & analysis

scripts/convert_pdf_to_images.py is provided for the fallback case where text extraction yields nothing useful (scanned PDFs).

---

License

MIT