Retrieval Layer

Requirements — second-brain retrieval layer v2

At a glance

• What it is: upgrade search_vault from semantic-only search to a schema-aware hybrid retrieval layer — hard filters + semantic ranking in one query, schema exposed to the agent, scoped read-only endpoints
• Status: Draft v2 — R-1 resolved after vault-indexer code review; Phase 0 added, frontmatter filter descoped
• Depends on: vault-indexer (metadata extraction — requires Phase 0 changes below), second-brain v1 (common.py, embed.py, mcp_server.py)
• Serves: Job A

---

1. Background and problem statement

The current MCP server exposes search_vault(query, top_k) — pure cosine similarity over Ollama embeddings. Three limitations:

1. No structured filtering. The vault index already contains metadata (path, folder, tags, frontmatter, modified date), but none of it is queryable through the tool. A query like “homelab notes from the last quarter about upgrades” can only be approximated semantically.
2. The agent is blind to the vault’s shape. Claude doesn’t know which folders, tags, or frontmatter keys exist, so it cannot construct precise queries and instead over-relies on semantic phrasing.
3. Results are thin. Wikilinks-only results force one follow-up file read per hit, multiplying round trips and context cost.

Additionally, there is no way to scope a server instance to a subset of the vault. The privacy note in the README (hook context is sent to Anthropic) applies equally to MCP results; a structural scope is the correct fix, not score thresholds.

2. Goals and non-goals

Goals

• G1. Single-call hybrid retrieval: metadata filters applied before semantic ranking.
• G2. Schema-aware tool descriptions generated from the live index at server startup.
• G3. Result payloads rich enough that one search call usually suffices.
• G4. Per-instance scoping (path filter + custom instructions), composable with the existing multi-vault --db/--name mechanism.
• G5. Zero new infrastructure at Layer 1 — SQLite + Ollama only. pgvector (Layer 2) remains a drop-in backend swap, not a requirement.

Non-goals

• NG1. No write operations. The server stays read-only by design.
• NG2. No changes to embed.py’s embedding model or chunking strategy (separate workstream).
• NG3. No changes to Mode 1 (hook) behavior in this phase, except where shared code in common.py benefits it for free.
• NG4. No reranker/LLM-in-the-loop retrieval. Cosine + filters only.

3. Current state (v1 contract, to be preserved)

• Tools: search_vault(query, top_k=5) → wikilinks; get_vault_stats() → usable coverage report.
• Config via env: SECOND_BRAIN_OLLAMA_URL, SECOND_BRAIN_EMBED_MODEL, SECOND_BRAIN_DB.
• Embeddings keyed by path; search_vault inner-joins to files, silently dropping stale rows; embed.py prunes orphans.
• Tests run without Ollama and without the mcp package installed.

Compatibility requirement: v1 calls (search_vault with only query/top_k) must behave identically in v2.

4. Functional requirements

FR-0x — Phase 0: vault-indexer prerequisites

Code review resolved R-1: tags are parsed by extract_sections but persisted only inside the extracted JSON TEXT column, which cannot support efficient tag-AND WHERE clauses. The following additive changes to vault_indexer.py are prerequisites for FR-11:

• FR-01 — New table tags(path TEXT NOT NULL, tag TEXT NOT NULL, PRIMARY KEY(path, tag)) with an index on tag. Created in open_db following the existing migration pattern (cf. the content_hash / content_mtime migration).
• FR-02 — scan_vault writes tag rows whenever it writes/updates a file row: delete existing rows for the path, insert current tags. On a detected move, re-key rows from the old path to the new one (or delete+reinsert). Tags come from the same extract_sections output already computed — no extra file reads.
• FR-03 — One-time backfill for existing DBs: on migration, populate tags from the stored extracted JSON of every row, so a full --reindex is not required to start filtering.
• FR-04 — The extracted column, summaries, and all existing vault-indexer behavior are unchanged. Existing consumers must not notice Phase 0.
• FR-05 — Rows in tags whose path has no matching files row are treated exactly like stale embeddings: excluded by join, pruned opportunistically.

FR-1x — Hybrid search

• FR-10 — search_vault accepts optional filter parameters in addition to query and top_k:
  • folder (string, prefix match on path, e.g. "01-Projects")
  • tags (list of strings; a result must match all given tags — AND semantics)
  • modified_after / modified_before (ISO date strings, matched against content_mtime so pure moves don’t register as activity) Descoped from v2: a generic frontmatter key→value filter. extract_sections discards all frontmatter except tags, and the vault’s prefix-tag convention (type/…, created/…, status/…) already encodes structured metadata as tags — so tag filters cover the use case without new extraction work. Revisit only if a real need appears that prefix tags can’t express (see Q-2).
• FR-11 — Filters are applied as SQL WHERE clauses against the files and tags tables (the latter added in Phase 0) before embeddings are loaded and ranked. Rationale: correctness (top-k of the filtered set, not filtered top-k) and performance (fewer vectors in memory).
• FR-12 — query becomes optional when at least one filter is present. Filters-only calls return results ordered by modified date descending (no cosine pass, no Ollama call).
• FR-13 — An empty filter result returns an explicit “0 matches for filters: …” message (not an empty list), so the agent can distinguish “nothing relevant” from “filter too narrow” and self-correct.
• FR-14 — Invalid filter values (malformed date, tag that exists nowhere in the index) return a structured error naming the invalid parameter and, where cheap, the valid options. Never a stack trace.

FR-2x — Schema-aware tool description

• FR-20 — At startup, the server queries the index and generates the search_vault tool description containing: top-level folder names, the N most frequent tags (default N=30), available frontmatter keys with an example value each, file count, and date range of the corpus.
• FR-21 — The schema block is compressed prose/lists, budgeted to roughly 150–300 tokens. It is a map, not a dump.
• FR-22 — Schema generation must complete in under ~200 ms against a 5,000-file index (it runs once per server start; the budget guards against pathological queries).
• FR-23 — If schema generation fails (missing tables, empty index), the server starts anyway with the v1 static description and logs a warning. Schema is an enhancement, never a startup dependency.
• FR-24 — When a scope filter is active (FR-4x), the schema reflects only the scoped subset — the description must not leak folder/tag names from outside the scope.

FR-3x — Result payload

• FR-30 — Each result returns: wikilink (unchanged format), similarity score, folder, tags, modified date, and a snippet.
• FR-31 — Snippet = the first ~400 characters of body, frontmatter stripped. Configurable via SECOND_BRAIN_SNIPPET_CHARS. Neither vault-indexer nor embed.py stores body text, so snippets are produced by reading the file at query time — acceptable for ≤top_k local reads per call, and inherently stale-proof (a snippet always reflects the file on disk, not the index). If the file no longer exists at the indexed path, return the result without a snippet and a (file moved — reindex) marker rather than erroring.
• FR-31a (future, depends on NG2 chunking workstream) — When chunk-level embeddings exist, snippet = best-matching chunk. Not required for v2.
• FR-32 — A search_vault response is capped (default ~4,000 tokens worth of text); if top_k × snippet size would exceed it, snippets shrink before results are dropped.
• FR-33 — New tool read_note(wikilink_or_path) returns full note content, so the search payload can stay lean while deep dives remain one call away. Read-only; refuses paths outside the vault root and, when scoped, outside the scope.

FR-4x — Scoped instances

• FR-40 — New CLI flag --scope <glob-or-prefix> restricts every tool (search, read, stats) to matching paths. Enforced in SQL/path checks server-side — not by prompt instructions.
• FR-41 — New CLI flag --instructions "<text>" (or --instructions-file) appends operator guidance to the server/tool description (e.g. “Only surface work notes. Answer in English.”).
• FR-42 — Scope composes with existing --db and --name, enabling e.g. a vault-search-work instance scoped to 01-Projects/** alongside an unscoped personal instance.
• FR-43 — register.py gains support for writing scoped server entries into .mcp.json (additive merge policy unchanged, dry-run by default).
• FR-44 — get_vault_stats reports the active scope and counts within it, so a misconfigured scope is visible instead of silently returning thin results.

5. Non-functional requirements

Non-functional requirements

• NFR-1 Latency: hybrid search p95 < 1.5 s on Layer 1 (5k files, laptop), excluding Ollama embedding latency for the query itself.
• NFR-2 Memory: filtered vector loading must not regress the current full-load worst case; with filters it should load strictly fewer vectors.
• NFR-3 Robustness: dead Ollama degrades to filters-only mode with a warning in the result, mirroring the hook’s fail-silent philosophy.
• NFR-4 Portability: no new required dependencies. mcp remains optional for the test suite; SQL stays SQLite-compatible, with the pgvector variant isolated behind the same interface in common.py.
• NFR-5 Privacy: scoping is a hard boundary. No tool, including error messages and stats, may reveal content or names from outside the active scope.

6. Interface specification

search_vault(
  query: str | None = None,
  top_k: int = 5,
  folder: str | None = None,
  tags: list[str] | None = None,
  modified_after: str | None = None,    # ISO 8601, vs. content_mtime
  modified_before: str | None = None,   # ISO 8601, vs. content_mtime
) -> list[Result]

Result = {
  wikilink: str, score: float | None,   # None for filters-only calls
  folder: str, tags: list[str],
  modified: str, snippet: str,
}

read_note(target: str) -> str            # wikilink or vault-relative path
get_vault_stats() -> str                 # + active scope, scoped counts

New environment variables (defaults preserve v1 behavior):

Variable	Default	Purpose
SECOND_BRAIN_SNIPPET_CHARS	400	Fallback snippet length
SECOND_BRAIN_SCHEMA_TAGS	30	Tags listed in schema block
SECOND_BRAIN_RESULT_BUDGET	4000	Approx. token cap per response

7. Implementation phases

Waterfall-friendly ordering; each phase is releasable.

0. Phase 0 — vault-indexer prerequisites: tags table, scan-time writes, migration + backfill (FR-0x). Lands in the vault-indexer repo/folder with its own tests; second-brain v2 declares a minimum index schema version.
1. Phase 1 — Query core: filter → rank pipeline in common.py (pure function over the DB, no MCP dependency), including FR-12/13/14 semantics. This is where most tests live.
2. Phase 2 — Tool surface: extend mcp_server.py signatures, add read_note, result payload shaping and budget.
3. Phase 3 — Schema generation: startup schema block + failure fallback (FR-2x).
4. Phase 4 — Scoping: --scope/--instructions, register.py support, scoped stats and schema (FR-4x, NFR-5).
5. Phase 5 — Docs: README update (tool table, hook-vs-MCP table gains a “filters” row), CHANGELOG entry, multi-vault guide update.

8. Testing requirements

Extend the existing suite; same constraints (no Ollama, no mcp package, in-memory SQLite fixtures).

Testing requirements

• T-1 Filter semantics: each filter alone, combined (AND across types), tags-AND within, prefix vs. exact path matching.
• T-2 Order of operations: verify top-k over the filtered set (construct a fixture where filtered top-1 ≠ global top-k members).
• T-3 Filters-only path performs no embedding call (assert via stubbed Ollama client).
• T-4 Empty-result and invalid-filter messages (FR-13/14).
• T-5 Stale-path rows remain excluded under every filter combination (regression guard for the folder-reorg incident).
• T-6 Schema block: content, token budget, scoped subset only, and graceful fallback on a broken index.
• T-7 Scope enforcement: read_note path traversal attempts, out-of-scope wikilinks, stats leakage.
• T-8 v1 compatibility: search_vault(query, top_k) byte-for-byte-equivalent wikilink output on a fixture vault.
• T-9 Phase 0 (vault-indexer suite): tag rows written on new/updated files; re-keyed on move; backfill populates from existing extracted JSON; orphaned tag rows excluded by join.
• T-10 Snippets: frontmatter stripped, SECOND_BRAIN_SNIPPET_CHARS respected, missing-file marker instead of error, no snippet reads beyond the returned result set.

9. Acceptance criteria

Acceptance criteria

• AC-1. “Kubernetes notes tagged homelab modified since June, most relevant to ‘upgrade rollback’” resolves in one tool call with usable snippets.
• AC-2. A fresh Claude Code session, without being told anything about the vault, constructs a valid filtered query on first attempt (schema block is doing its job).
• AC-3. A --scope 01-Projects/** instance cannot be induced — by search, read, stats, or error text — to reveal anything outside that prefix.
• AC-4. Full test suite passes on a machine with neither Ollama nor mcp installed.
• AC-5. v1 clients and the Mode 1 hook work unchanged.

10. Risks and open questions

Risks and open questions

• R-1 (resolved → Phase 0) Tag storage cannot support efficient AND filters: tags persist only inside the extracted JSON column. Resolution: tags junction table written during scan (FR-0x). Additive; no consumer changes.
• R-2 (resolved → FR-31) No body text is stored anywhere in the index, and v1 embeds whole files. Resolution: read files at query time for snippets (stale-proof, cheap locally); best-chunk snippets deferred to the chunking workstream (FR-31a).
• R-3 Glob scoping (**) vs. simple prefix: globs are friendlier but easy to get subtly wrong in SQL. Recommendation: ship prefix-only, name the flag --scope, document glob as future work.
• R-4 Phase 0 backfill reads the extracted JSON of every row once; on a corrupted row, skip and warn rather than abort the migration.
• Q-1 Should --instructions also apply to the hook (Mode 1)? Deferred; hook scoping is listed in the README as a separate concern.
• Q-2 (reframed after frontmatter descope) If structured metadata beyond tags is ever needed for filtering, the options are (a) extend the prefix-tag taxonomy — zero code, or (b) extend extract_sections to persist a flat frontmatter map + junction table. Prefer (a) until it demonstrably fails.

11. Appendix — content conventions that make retrieval work

Content conventions

The tooling has no hard schema, but extraction quality tracks these conventions:

1. Frontmatter must start at byte 0 — FRONTMATTER_RE anchors at the first line; anything before --- silently disables tag extraction for that file.
2. Tags live in frontmatter, inline (tags: ["#a"]) or list form; body #tags are invisible to the index.
3. Prefix tags are the metadata system: type/, created/YYYYMMDD, and any future area/, status/ prefixes. Keep the taxonomy small and low-cardinality — filters and the schema block (FR-2x) are only as useful as the tags are consistent.
4. One H1 (becomes the title), meaningful H2s (surface in summaries; future chunk boundaries).
5. Front-load the first two paragraphs with what the note is about — that’s what whole-file embeddings weight most and what the FR-31 snippet shows.
6. Date-prefixed folders (YYYYMMDD-…/) for work items — they drive grouping and range-summary dates.