Building a Local “Second Brain” Serendipity Miner: Books, Tweets, and Web Clippings

By | November 20, 2026

Building a Local Second Brain Serendipity Miner

Most read-it-later workflows are write-only graveyards. You save an interesting article, drop it into a folder, and never look at it again—even when your own notes already hold a book you read two years ago or a thread you liked that connects directly to it.

If you have spent years accumulating a personal knowledge base in Obsidian—book catalogues, exported social archives, project drafts, and technical notes—you don’t need another cloud bookmarking SaaS. You need a Serendipity Miner that links every new URL to the knowledge you already own.

Here is how we built ~/scripts/serendipity_miner.py and wired it into a <1s-feeling mine <url> command inside our Zero-Trust Signal Gateway.


Architecture: Extract, Score, Resonate, File

Second Brain Serendipity Miner Architecture

The pipeline runs completely locally in four stages without requiring an LLM round-trip just to file a link:

  1. Zero-Friction Capture (mine <url>): Send any URL to Signal “Note to Self” (or run serendipity_miner.py from the CLI).
  2. Clean Extraction & Concept Scoring: Strips navigation, scripts, and boilerplate into clean Markdown, then ranks the top salient keywords.
  3. Parallel Cross-Corpus Search (ThreadPoolExecutor): Queries your local vault index across four distinct silos simultaneously.
  4. Obsidian Filing + Blog Seed: Writes a structured note under 03-Resources/Clippings/ with real Obsidian wikilinks to matching books, tweets, and projects.

1. Clean Markdown Extraction with Graceful Fallback

When defuddle is available on PATH, the miner delegates parsing to defuddle parse <url> --md. When running on headless nodes without Node tooling on PATH, it falls back transparently to BeautifulSoup4 + html2text:

for tag in soup(["script", "style", "nav", "footer", "aside", "noscript", "svg", "form"]):
    tag.decompose()

main_el = soup.find("article") or soup.find("main") or soup.body or soup
converter = html2text.HTML2Text()
converter.body_width = 0
converter.ignore_images = True
md_text = converter.handle(str(main_el)).strip()

This keeps extraction fast, deterministic, and resilient across arbitrary blogs and documentation sites.


2. Deterministic Concept Extraction (Zero-LLM Latency)

To search your vault without waiting 10 seconds for an LLM to summarize the page first, extract_keywords() weights tokens by structural position:

  • Title tokens: +8
  • Meta description: +4
  • H1–H3 headings: +3
  • Body sample (first 6,000 chars): +1

After filtering English and technical boilerplate stopwords, the top 4–6 keywords form a high-signal search query in under 5 milliseconds.


3. Parallel Resonance Across Books, Twitter/X, Knowledge, and Projects

Instead of running one global vault search—which gets dominated by recent logs—we fan out four scoped searches in parallel using ~/scripts/search_vault.py --folder:

CORPORA = [
    ("Books", "03-Resources/Books"),
    ("Twitter/X", "03-Resources/Twitter"),
    ("Knowledge", "Knowledge"),
    ("Projects", "01-Projects"),
]

Running these four queries concurrently via ThreadPoolExecutor(max_workers=4) takes ~100–250ms total and guarantees that an old book in your library or a tweet you liked in 2021 isn’t crowded out by yesterday’s session notes.


4. Filing with Vault Resonance and Blog Seed Callouts

The resulting note in 03-Resources/Clippings/<YYYY-MM-DD> - <Title>.md opens with an Obsidian > [!INFO] Vault Resonance callout containing clickable wikilinks grouped by corpus. Whenever 2 or more corpora match, it automatically appends a > [!TIP] Blog Seed callout highlighting the cross-domain overlap.

In Signal, the reply stays mobile-compact:

💎 Serendipity Mined: Self-Compacting & Bi-Temporal Agent Memory
• Keywords: memory, context, phase, engineering, bi-temporal (bs4+html2text)
• Saved: 03-Resources/Clippings/2026-11-18 - Self-Compacting & Bi-Temporal Agent Memory.md
• Vault Resonance (12 match(es)):
  ↳ Books: Political Trust..., Wristwatch Annual...
  ↳ Twitter/X: Liked Tweets 01, Liked Tweets 07
  ↳ Knowledge: Regression-gated nightly memory...
  ↳ Projects: Self-Compacting and Bi-Temporal Agent Memory

Paired with Self-Compacting & Bi-Temporal Agent Memory, every article you read immediately weaves itself into your existing graph instead of sitting isolated in a bookmarks folder.


Key Takeaways

  • Scoped parallel searches beat global search: Querying Books, Twitter/X, Knowledge, and Projects separately surfaces unexpected cross-domain connections that global ranking hides.
  • TF-weighted structural keywords eliminate LLM cold-start: Title + heading boosting extracts accurate search terms in milliseconds.
  • Mobile capture closes the loop: Wiring mine <url> into Signal turns passive scrolling into structured, backlinked Obsidian notes.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.