Source ingest pipeline

This page defines the lightweight first-pass ingest loop for Shanepedia.

Goal

Turn a raw source into a durable wiki artifact with minimal friction:

• raw source exists under raw/sources/
• source summary page gets created in wiki/sources/
• log is updated
• index is refreshed
• lint is run

Command

python3 scripts/wiki/ingest_source.py raw/sources/.../file.md \
  --title "Descriptive title" \
  --summary "One short synthesis paragraph." \
  --why "Why this matters for Shanepedia or Shane." \
  --related ../ops/shanepedia-agent-runtime.md \
  --related ../topics/llm-wiki-pattern.md \
  --suggest-related \
  --suggested-related ../projects/shanepedia-studio.md

What it does

1. validates the raw source path
2. creates a first-pass source page under wiki/sources/
3. appends an ingest entry to wiki/log.md
4. rebuilds wiki/index.md
5. runs lint checks

Suggested related pages

Use --suggest-related when you want the ingest command to scan the existing wiki and record likely write-back targets.

• confirmed navigation stays in related
• suggested follow-up targets are stored in suggested_related
• the created source page also includes a ## Suggested related pages section for quick manual review
• use --suggested-related to pin extra follow-up targets even if the auto-suggester misses them

This keeps the workflow markdown-first: the source page itself becomes the durable record of what likely needs to be updated next.

Operating note

This is intentionally a lightweight first pass. Good ingest still requires agent judgment:

• update affected topic/entity/project pages
• review and prune the suggested related pages list
• strengthen maps or dashboards if the source changes navigation
• write back reusable insights beyond the source summary itself

Best use

Use this when a new source arrives and you want the wiki to start absorbing it immediately instead of waiting for a full manual pass.