Skip to content
obsidian-wiki

The ingest pipeline

A source dropped into an entry point goes through seven steps, from a hash check that skips unchanged files to writing a page and updating the tracking files. Nothing runs on its own; the pipeline runs when you invoke /ingest.

INGEST ONE SOURCE HASH MATCH file dropped in entry point compute SHA-256 compare to manifest skip no-op classify source assign source_quality extract knowledge items discard low-signal route per CLAUDE.md structured-knowledge folder write or update page frontmatter, provenance post-ingest rule move or keep update manifest, log, hot.md append + push pipeline step write page (focal) skip branch

The seven steps

Section titled “The seven steps”
  1. The agent computes SHA-256 and compares with _service/.manifest.json. If the hash matches, it skips; reruns never re-process unchanged files
  2. Classify the source by type and assign a source_quality score from a fixed bucket list (paper, official, documentation, article, blog, voice-transcript, claude-chat, and so on)
  3. Extract knowledge items: entities, claims, links. Discard greetings, dead-ends, and low-signal content
  4. Route each item to a knowledge folder per the routing rules in CLAUDE.md
  5. Write or update the page with full frontmatter: summary (≤200 chars), sources, base_confidence, lifecycle: draft, provenance fractions. Apply inline provenance markers (^[inferred], ^[ambiguous]) on individual claims
  6. Apply the entry point’s post_ingest rule: either add processed: true and move the file under _service/entry-points/<entry-point>/<YYYY-MM>/, or add the frontmatter and leave it in place
  7. Update the manifest, append a one-liner to _service/log.md, push the touched page onto _service/hot.md

Minimum page size

Section titled “Minimum page size”

The minimum page size is 250 words. If a knowledge item cannot reach that threshold, the agent defers it until more material accumulates rather than producing a stub page.

Section titled “Related reference”