lode — External sources & the knowledge graph¶

(§6) The AI annotation layer gains read access to external sources — tickets, source repos, wikis, email — and draws down explicitly linked web pages, integrating everything into the knowledge graph. This is added incrementally, after the core loop works (design.md §7). The retrieval that runs over this graph lives in retrieval.md.

Externals fit the same model¶

A fetched ticket / wiki page / email / web page is structurally a note version: an immutable, point-in-time snapshot. The whole store collapses to one shape:

immutable content nodes (some owned = your notes, some mirrored = externals) + a derived annotation/link layer + head pointers.

Axes on a content node: origin: owned | mirrored; on derived items: source: ai | user.

Diagram

The broken assumption: external staleness is NOT topological¶

For owned notes, staleness is free (head moved → you know instantly). For externals, the true head lives on someone else's server and changes without telling you. Consequences:

Externals need a refresh policy (TTL / on-access revalidation / webhook) — there is no structural staleness signal. (Per-connector judgment; see decisions.md. Decided for the web connector: a scheduled TTL sweep — see Refresh policy below.)
Every AI claim from an external must cite "as of fetched_at." "The ticket is open" is a lie; "the ticket was open as of last sync, 3 days ago" is honest.
Retrieval uses an explicit trust gradient, in both ranking and citation display: your note > your annotation > current external snapshot > stale external snapshot > AI-inferred edge. The user's own words are highest-trust; externals corroborate, they do not override. This is the trust_rank step in retrieval.md.

External identity — same two-id split¶

external_id — stable logical identity: JIRA-1234, repo@path@commit, email Message-ID, normalized URL.
snapshot_id — immutable fetched version (content hash).
One canonical node per external_id with many edges — never five copies of a ticket linked from five notes. Dedup on external_id; version on snapshot_id.

A query result has no identity — discovery is not citation (decided, `lode-35nu.11.5`)¶

Tool-augmented Ask (lode-35nu.11) lets the LLM reach live external systems mid-answer, and the settled constraint there is that every cited tool result is persisted as an external snapshot before synthesis, so the faithfulness gate verifies spans against stored bytes exactly as it does for any other external. That constraint collides with this section: external_id is a primary key that is either a fetchable URL (web, above) or a semantic key — issue key / page id (Atlassian, lode-gpzn.2, with api_base carrying the rebuild base). "JIRA search for X" is neither.

Both obvious escapes are wrong. Minting an identity for query results — content-addressing the result set, or query-addressing it — fails on the same fact from two directions: a query's results change underneath you, so content-addressing churns a fresh primary key on every run (defeating the one-node-per-source dedup this whole section exists to guarantee), while query-addressing yields one durable row whose content silently mutates (defeating snapshot_id's immutability). And banning query tools outright would leave the model able to fetch only what it already knows the key of, which is most of the value gone.

So the resolution is neither: split discovery from citation.

A search/query tool is allowed, and returns only identifiers and titles — never body text. Its output is never written to externals, never given a snapshot_id, and is never a citation target. It is navigation, not evidence. Returning no body text is not a nicety — it is the mechanism: there is nothing in a search result the model could quote, so the faithfulness gate cannot be routed around by citing a search response.
A fetch tool then retrieves the specific resources those identifiers name. Those are addressable — their external_ids are exactly the ones this section already defines — so they land on the existing draw-down path with a valid primary key, and are cited by snapshot_id like anything else.

The consequence worth stating plainly: no new identity scheme is introduced, and the faithfulness gate is untouched. The question "what is the external_id of a search?" is not answered — it is deleted, by never persisting a search.

Ask-time snapshots are first-class, with a provenance marker (decided, `lode-35nu.11.5`)¶

Because of the rule above, every snapshot the Ask path writes is a snapshot of a real addressable resource — byte-for-byte the same object lode.drawdown already produces when a note links that ticket. Giving those rows a separate "ephemeral" class with its own lifecycle and GC would mean maintaining a second lifecycle and a second code path to distinguish rows that are structurally identical, so they are ordinary externals/snapshots rows, visible to Browse, reconcile, staleness scanning and re-embedding like any other.

Two qualifications keep that from being a silent corpus mutation:

discovered_via = 'ask' is recorded on the externals row, so Browse can filter and the origin of a row is never a mystery. It is a provenance marker only — nothing branches on it.
No note→external edge is created. Asking a question is not an assertion that the answer's sources belong to any note, so nothing links these rows into the graph. They are reachable and retrievable; they are not claimed by a note.

The dedup requirement this raises is already satisfied by existing machinery, with nothing new to build: repeated asks about the same resource dedup on external_id (one node), and an identical refetch is free because snapshot_id = H(external_id ‖ body) yields the same hash and no new row.

URL canonicalization (decided, `lode-w0h.3`; userinfo stripped, `lode-0as`)¶

For a web source, external_id is its canonical URL string — not a hash — so this canonicalization is the dedup correctness "same URL in two notes = one node" depends on, and the same join key lode-w0h.6's refresh policy reuses to find "the same source" across refetches. Applied in order (lode.drawdown.canonicalize_url):

Lowercase the scheme and host (path/query stay as-cased — some servers treat them case-sensitively).
Drop userinfo (user[:pass]@) entirely. Credentials in a pasted URL are transport secrets, not source identity — the same reasoning that strips utm_* below, applied to the strongest instance of that class. https://user:pass@host/p, https://user@host/p, and https://host/p all canonicalize to one external_id.
Strip the port when it equals the scheme's default (:80 for http, :443 for https).
Drop the fragment (#...) entirely — never part of server-side identity.
Strip query params matching the tracking blocklist (url_tracking_param_blocklist, configuration.md; default utm_*, fbclid, gclid).
Sort the remaining query params.
Normalize the trailing slash: an empty or bare / path becomes /; any other path loses a trailing /.

Why userinfo is a privacy fix, not a dedup nit (lode-0as): external_id is a durable identifier — it is a primary key, it is replicated into edges.to_id on every citing note, and it is reachable from the retrieval candidate set. Hard delete scrubs versions.body; it does not scrub an identifier already copied into edges and indexes. A credential that reaches external_id therefore outlives the content it guarded and survives a delete that looks complete. Stripping it at canonicalization time is the only point where the secret can be kept out of storage entirely, rather than redacted after the fact.

Two adjacent gaps, decided out of scope here (lode-0as): path percent-encoding (%7E vs ~) and IDN hosts (unicode vs punycode) are not normalized, so two URLs that are "the same page" in those respects still canonicalize to distinct external_ids and dedup as two nodes. This is a deliberate decision, not an oversight: unlike userinfo, neither is a privacy surface — both are ordinary dedup-correctness gaps, and normalizing either safely needs care (percent-decoding a path requires respecting RFC 3986 reserved-vs-unreserved characters so a literal %2F isn't mistaken for a path separator; IDN punycoding needs a real IDNA implementation) that this ticket did not need to take on to close the privacy hole. Left for a future ticket if a real corpus ever shows it matters.

Migration note (lode-0as, 2026-07-09): changing canonicalize_url changes external_id for any web source already stored under the old (userinfo-preserving) form — a re-key + edge-repoint would be required for existing rows (the same shape as _repoint_edges, used today for the redirect wrinkle below). Checked at the time of this fix: no externals rows and no edges.to_id containing userinfo existed anywhere, so no migration was run. If this is ever revisited against a store with real web draw-down history, check first.

The redirect wrinkle: a note's edge is created against the pasted URL's canonical form before any fetch runs (fetching is the queued async job). If the fetch follows a 3xx chain to a different final URL, the snapshot is ingested under the final URL's canonical external_id instead, and every source='user' edge asserted against the pasted-canonical id is re-pointed onto the final-canonical id — so the note's edge always resolves to the node that actually holds content, and a persistently-redirecting URL still dedups onto one node.

Snapshot churn: decouple new snapshot from re-enrich¶

snapshot_id = H(external_id ‖ body) makes an identical refetch free (same hash, no new row). But a chatty external — an active PR refreshed hourly, one new comment each time — produces a new snapshot every refresh, and naively each one would trigger a paid re-enrichment call through the enrichment LLM (stack.md batch path). Owned notes have a no-op-save guard; externals need the analogous cost control, so the two operations are gated separately:

Re-embed on any change — local, cheap, keeps retrieval current.
Re-enrich only on material change — gate the expensive enrichment on a local delta between the new snapshot and its predecessor; below the bar, carry the prior enrichment forward, re-anchored to the new snapshot. A one-comment PR update re-embeds but does not re-enrich.

Materiality signal (lode-w0h.5, pinned after debate): embedding-similarity delta alone, not size — cosine similarity between the new and predecessor snapshot's mean-pooled passage vectors, computed post-embed (the signal needs the new snapshot's own vectors, so the gate runs from lode.worker._embed_handler right after lode.embedding.embed writes them, not from ingest_snapshot at write time). delta = 1 - similarity; material iff delta >= reenrichment_materiality_threshold. No predecessor to compare against — the external's first-ever snapshot, or a predecessor that was never embedded (e.g. a tombstone) — is unconditionally material: there is nothing to carry forward either. Below the threshold, lode.externals.gate_reenrich enqueues no enrich job and instead carries the prior enrichment forward by re-anchoring it (lode.staleness.reanchor_annotations / reanchor_edges, targeting the external_id) — the identical quoted-text mechanism Repository.save already runs for a note update, not a hand-rolled copy of the rows. At/above the threshold, one enrich job is enqueued for the new snapshot_id.

The materiality threshold is a tunable knob (configuration.md); it caps cloud spend on noisy sources without letting enrichment rot.

The enrich job the gate enqueues resolves polymorphically (lode-7qi): lode.enrich's three entry points (enrich_version, and the Batches API route submit_enrich_batch/collect_enrich_batch that actually claims a pending enrich job first in production) all resolve target_version against versions/notes first, falling back to snapshots/externals — the same blind resolution lode.embedding._version_body already used for the embed leg. A material change therefore runs a real enrichment-LLM extraction (default: Haiku) over the snapshot body and writes annotations/edges against the external_id (the same polymorphic annotations.target / edges.from_id a note's note_id writes against).

The write path (lode.externals.ingest_snapshot, lode-w0h.2) is the mirrored analogue of the note save path: dedup on external_id (one externals row per source, created on first sight), compute snapshot_id and skip the write entirely when it equals the current head (the identical- refetch-is-free case above), otherwise insert the new snapshot and move head_snapshot_id. An ok snapshot then enqueues embed (never enrich — that gate is lode-w0h.5's, decided post-embed) and drives the synchronous FTS leg itself (lode-c5l, rebuild of the bounced lode-w0h.8) — the same cache-after-commit shape Repository.save uses for owned notes, so an ingested snapshot is keyword-findable the instant ingest_snapshot returns, before the async embed worker ever runs. A fetch failure (Draw-down rules below) writes a tombstone snapshot whose body is the stable, inspectable marker "[tombstone: <reason>]" — itself content-addressed, so a source that keeps failing the same way dedups its tombstones too, rather than growing one row per retry — but enqueues no embed job and writes no FTS row: a failed fetch must not become a retrievable/citable hit on either leg (decision, lode-w0h.2, 2026-07-08, extended to the FTS leg by lode-c5l; mirrors the owned-note delete path, which likewise is never indexed).

Refresh policy: TTL-based revalidation (decided-for-web, `lode-w0h.6`)¶

decisions.md's "External refresh" entry leaves each connector to choose between on-access revalidation and scheduled background refresh, both with an eye toward a TTL. For the web connector: scheduled TTL sweep, not a true on-access hook.

Why not on-access. A true on-access hook would trigger revalidation the instant a citation reads a stale external — but every synchronous read path in this codebase is deliberately network-free (the same rule Repository.save follows on the write side: no network I/O inline). Bolting a blocking HTTP fetch onto an interactive Q&A call would trade a bounded, predictable citation latency for an unbounded one (a slow or hanging origin server would stall the answer that happens to cite it). No such hook exists on the retrieval/Q&A read path today, and building one is out of this ticket's "staleness detection + scheduling only, no second fetch path" scope.

The policy: ride the reconciliation scan. lode.reconcile's refresh_stale step (lode-w0h.6) runs alongside embed_gap/enrich_gap — at worker startup and every --loop/--wait poll tick (docs/storage.md "Reconciliation scan on startup + periodically"). Each pass: find every external whose current head snapshot is non-tombstone and older than refresh_ttl_s (configuration.md, default 1h) with no refresh job already pending/running, and enqueue one via the same lode.jobs.enqueue_derive_jobs path every other step uses (ON CONFLICT DO NOTHING against idx_jobs_live — idempotent, safe to run any time or frequency). The enqueued job is handled by the exact same lode.drawdown.refresh_external (lode-w0h.3) the paste-triggered first draw-down uses — this ticket adds no second fetch path, only the staleness check and the enqueue. A changed body re-embeds and re-enriches-if-material exactly as any other new snapshot does (Snapshot churn above); an unchanged refetch is free (identical snapshot_id, no new row).

In practice this reads as "revalidate the next time anything drains the queue, if the TTL has elapsed" rather than "revalidate the instant a citation reads it" — a coarser cadence than true on-access, but one that never adds latency to a read, and matches decisions.md's own leaning ("on-access with a short TTL cache ... for a single instance with finite API quota") in spirit: bounded, cheap, TTL-gated re-fetching, without the network-on-the-read-path cost on-access would actually require.

Tombstones are excluded from the sweep, deliberately. A tombstone head means the source already failed permanently — either a genuine 4xx/empty-extract, or a transient failure that exhausted every retry and dead-lettered (Fetch-outcome taxonomy above; lode-at8's dead-letter hook is what writes that tombstone). Re-fetching it on every TTL sweep would burn queue churn on a source the draw-down machinery has already classified as unfetchable-for-now — the same reasoning lode.reconcile's embed_gap step already applies (a tombstone has no body worth re-embedding either). If a dead link recovering later ever matters in practice, nothing structural prevents it — a user re-pasting the same URL still triggers a fresh paste-time draw-down (lode-w0h.3) regardless of the sweep. Revisit if this proves too conservative.

Join key: canonicalize_url's narrow canonical form, confirmed correct, not silently inherited. This step's staleness/scheduling join key is externals.external_id — the same canonical URL string URL canonicalization above defines. That section already settles, explicitly and by design, that path percent-encoding (%7E vs ~) and IDN hosts (unicode vs punycode) are not normalized — two "same page" URLs differing only in those respects are, and remain, two distinct external_ids that this policy schedules and refreshes independently, never recognizing them as one source. lode-w0h.6 confirms this narrow canonical is correct for the refresh policy too, for the same reason lode-0as gave it: normalizing either safely needs care (RFC 3986 reserved-vs-unreserved handling for percent-decoding; a real IDNA implementation for punycoding) this connector does not need to take on, and neither gap is a privacy surface the way userinfo was (userinfo is stripped — lode-0as — so no credential ever becomes part of this policy's join key or a refresh job's target_version). Widening the canonical form remains a live option for a future ticket, but would require a re-key + edge-repoint migration (lode-0as's migration note) — deferred, not silently assumed away.

A single default TTL, not yet per-source. refresh_ttl_s is one knob for every external (only web sources exist today). decisions.md's "a closed ticket rarely changes, an active PR changes hourly" framing is a real per-source judgment call a future connector (or a future per-source_type override) may want to make — deferred, not built, since there is exactly one connector and one judgment to make today.

Backfill: per-connector re-draw-down (`lode-gpzn.9`)¶

A connector's routing changes over time — the flagship case: a URL drew down through the generic web path (login page => tombstone, or a plain scrape) before a connector like Atlassian (lode-gpzn) existed or was flagged on, and should now route through that connector's structured fetch instead. lode backfill is the CLI command that re-runs draw-down for a connector's already-processed links under current routing — lode.backfill, src/lode/cli/backfill.py.

A framework, not a sweep. This ticket builds only the command + a registry seam each connector plugs its own backfill logic into (lode.backfill.register_backfill, mirroring lode.reconcile.register_step's shape) — it deliberately does not build one monolithic "detect everything at once" sweep, and ships no connector registered by default. lode backfill with no connector argument prints the command's own help; --list reports just the registered names. Neither runs anything (lode-6hi3). CLI only — no TUI surface, and it runs per-machine wherever the notes' DB lives ($LODE_HOME); it does not travel on the Dolt/git wire.

Shared plumbing, reused not reimplemented per connector: a connector's handler composes four pieces from lode.backfill — iter_user_linked_externals (walk every existing explicit source='user' note→external edge), mint_external (INSERT a fresh externals row for a newly computed identity, first-write-wins), repoint_edges (re-point source='user' edges old→new identity — reuses lode.drawdown._repoint_edges verbatim, the same function the redirect wrinkle above already rides), and enqueue_fresh_refresh (one fresh refresh job via the single shared enqueue path, lode.jobs.enqueue_derive_jobs). Each of mint/repoint/enqueue takes a dry_run flag, so lode backfill --dry-run reports what would change without writing anything.

Tombstone-exclusion override — re-run idempotency only (owner decision D, /challenge 2026-07-17). A per-connector backfill mints a brand-new, never-tombstoned semantic external on its first migration and enqueues a plain refresh — so on the first pass there is nothing tombstoned yet, and the override below never needs to fire. lode.backfill.needs_refresh mirrors the refresh policy's own s.status != 'tombstone' predicate by default (excluding a target whose head snapshot already tombstoned). The override matters only on an idempotent re-run where the new identity's own head snapshot already tombstoned on a prior backfill pass (e.g. a bad token => 401) — lode backfill --retry-tombstoned is the explicit, human-driven opt-in past that default, for the one case a periodic sweep structurally can't cover: an operator who just fixed the underlying cause and wants that specific already-tombstoned target retried now, not on a schedule.

jira connector (lode-gpzn.10). lode.jira_backfill registers "jira" — the first real connector plugged into the framework. lode backfill (src/lode/cli/backfill.py) registers it explicitly on every invocation, the same explicit per-invocation pattern confluence uses below (see lode.confluence_backfill.register's own docstring for why — not a bare module-level register_backfill(...) call at import time). Its handler reclassifies every existing explicit (source='user') edge's original quoted_text through lode.drawdown._classify_atlassian, the exact same synchronous, network-free classifier the live paste-time path uses — so a link migrates only when it would route to JIRA under current routing (flag on, credentials resolved, matched host, /browse/{KEY} shape). A match mints the semantic issue-key external, re-points the edge, and enqueues a fresh refresh through the four shared plumbing pieces above. Reclassifying from the edge's original quoted_text on every pass (rather than filtering on the edge's current source_type) is what makes --retry-tombstoned reachable on a re-run: an edge already repointed onto its semantic key is revisited too, so needs_refresh gets rechecked against that identity's current head snapshot instead of the migration step silently losing track of it after the first pass.

confluence connector (lode-gpzn.11). lode.confluence_backfill re-classifies every existing web-typed source='user' edge via lode.drawdown._classify_atlassian (the exact same detection detect_and_enqueue_drawdown already runs on a fresh paste, reused rather than re-derived) filtered to a Confluence-shaped match, then composes the four shared pieces above. Reusing _classify_atlassian also means the handler automatically honors the connector's own active-flag gate: with Confluence flagged off, no match is ever found and the backfill is a no-op, exactly like a fresh paste under the same settings. lode backfill (src/lode/cli/backfill.py) registers it explicitly on every invocation — not via a bare module-level register_backfill(...) call at import time, which would only ever fire once per process (Python's import caching) and cannot be relied on to survive an in-process test suite that deliberately clears the registry before each test. refresh_external's (lode.drawdown) SOURCE_TYPE_CONFLUENCE dispatch leg was wired separately by lode-mfts (mirroring lode-gpzn.3's identical JIRA-leg wiring), landing ahead of this connector.

Externals are directly retrievable¶

A snapshot's current head is a direct lexical/vector candidate on its own content, not only reachable via graph-expansion from a citing note: lode.retrieval.live_head_versions unions each external's non-tombstone head_snapshot_id alongside note heads, so both lexical_search and vector_search admit it (lode-c5l). A stale (non-head) snapshot stays excluded from both direct legs by construction — only head pointers are read. It is also unreachable via graph expansion: a graph_expand (lode-c4cd) note→external edge resolves to external_id, not a specific snapshot_id, so it always follows through to the external's current head — never a stale one — and trust_rank tiers a graph-reached external CURRENT_EXTERNAL accordingly, regardless of which edge type (user/ai) led there. lode.embedding.embed's body resolution is polymorphic (versions, then snapshots) so the embed job enqueued above runs to completion instead of raising.

Redact-before-index applies to both legs identically. The FTS leg chunks redact_before_index(body, settings), exactly what the vector leg's embed independently redacts — so a secret in a fetched page never lands in passages_fts/passages/LanceDB on either leg, and both legs chunk the same redacted text (needed for the embed worker's deterministic passage_id INSERT OR REPLACE to land on the same rows rather than orphan a stale one). snapshots.body itself is left untouched, exactly as versions.body is — see Two redactions, aimed at the right legs below.

Edges: explicit vs inferred¶

Explicit (a note cites JIRA-1234 or pastes a URL): high confidence, user-asserted edge. For a pasted web URL, this is lode.drawdown.detect_and_enqueue_drawdown (lode-w0h.3), called from Repository.save: it creates the source='user' edge and enqueues the refresh job that draws the page down — no network I/O in save itself, only the edge INSERT and the job enqueue, atomically with the version write.
Inferred (AI decides "the auth migration" is PR #42): a suggestion (source: ai, confidence-scored), never an asserted fact. Surface for confirmation; a user nod promotes it. This is where a hallucinated link would silently corrupt the graph — keep it gated.

Draw-down rules¶

Follow explicit links one hop, then stop. Pull the linked page, extract its entities, but do not follow that page's links outward. Recursion = unbounded web crawler, not a notes app. (This hop limit governs a fetched page's own outbound links; it is a separate knob from the HTTP redirect cap a single fetch follows — see the fetch-outcome taxonomy below.) Structurally enforced, not counted: only the note-save trigger ever scans note text for URLs (lode.drawdown.extract_urls); the refresh job handler (lode.drawdown.refresh_external, lode-w0h.3) never scans a fetched snapshot's body for further links.
Readability extraction + graceful failure. Many pages (JS-rendered, paywalled, 403) return scaffolding to a naive GET; strip nav/ads, snapshot cleaned text (+ optional raw HTML), and on failure write a tombstone snapshot rather than garbage.
Shared job type. The draw-down job reuses the refresh value already reserved on jobs.type — no schema migration. lode-w0h.3 introduces the fetch→ingest handler (lode.drawdown.refresh_external, registered in lode.worker); the paste-triggered initial draw-down is just the first refresh of a source, riding the same attempts/backoff/dead-letter machinery any later refetch would. lode-w0h.6's refresh policy reuses this handler unchanged and adds only staleness detection + scheduling.

Fetch-outcome taxonomy (decided, `lode-w0h.1`)¶

One fetch of one URL resolves to exactly one of these outcomes:

Outcome	Trigger	Result
OK	2xx response, extractor returns text at/above the length floor	snapshot with `status='ok'`
PERMANENT failure	401/403 and any other 4xx except 408/429; or a 2xx response whose extracted text is empty/`None`/shorter than the length floor (covers JS-rendered scaffolding, paywalled teasers, and empty pages with one signal); or a redirect chain longer than the configured cap (loop or merely-too-long — a retry hits the same cap)	snapshot with `status='tombstone'` — retrying will not help
TRANSIENT failure	408 or 429 (the two 4xx codes HTTP itself flags "try again later"), any 5xx, or a network/timeout error	not written as a snapshot by the fetch unit itself — the caller raises into the async work queue's existing attempts/backoff/dead-letter machinery (`failed` → `pending` retry, → `dead` at max attempts, PINNED `lode-i05.6`); on `dead`, the caller writes a tombstone snapshot so the note edge still resolves
3xx	one or more redirects, within the configured cap	followed transparently; the final resolved URL is what gets canonicalized into `external_id` — a note edge created on the originally-pasted URL may need re-pointing to the final URL's canonical form

The testable detection signal for "PERMANENT — 2xx but not real content" is the readability extractor returning None/empty or text below a configured length floor (configuration.md) — no separate paywall- or JS-shell-specific heuristic is needed. JS-rendered pages are a permanent tombstone by this rule, deliberately — actually rendering them (headless browser / JS execution) is an explicit deferred follow-on (lode-oni), not first-connector scope.

The HTTP-status half of this table is a shared, connector-neutral classifier (lode-gpzn.13). lode.fetch_outcome.classify_http_status is the single place the OK / TOMBSTONE / TRANSIENT status-code mapping lives — the web path (lode.webfetch) and the Atlassian connectors (JIRA lode-gpzn.3, Confluence lode-gpzn.4) all call it rather than each copying the 401/403/404 → tombstone, 408/429/5xx → transient rule. The extractor-driven "2xx but empty/short content" signal above stays connector-specific (trafilatura, in lode.webfetch) since it has no HTTP-status analogue. lode.worker._refresh_dead_letter_hook (the "on dead, the caller writes a tombstone" step just below) is generic over source_type too: it reuses an already-existing externals row's source_type rather than assuming web, falling back to web only when no row exists yet (today, only reachable for a web target whose first-ever fetch never succeeds).

A dead job's tombstone write must not beat an already-succeeded fetch (closed, lode-uda1 + lode-elc8). The TRANSIENT row's "on dead, the caller writes a tombstone" is unconditional only with respect to content that predates the dead-lettering job's own claim. lode.worker._reclaim_stale_running's crash-reclaim gate can dead-letter a refresh job that is not actually crashed, merely stalled past stale_running_timeout_s — and if its handler's own fetch then succeeds and commits a real snapshot before (or racing) the reclaim's dead-letter hook, the hook must not overwrite that real, current content with a tombstone. The hook is guarded on the job's claimed_at: it skips the tombstone write when the external's head is already a non-tombstone snapshot fetched at or after that claim (see docs/storage.md "A dead-letter hook's write can race a late success too" for the full race and rationale). This guard is orthogonal to — and neither depends on nor prejudges — the separate question of whether a late status='done' job-row write should itself be guarded (docs/storage.md "Crash reclaim: a job stuck in running"). The check and the tombstone write are now atomic (lode-elc8): the head read happens inside ingest_snapshot's own transaction, after it has already taken SQLite's write lock, rather than as a separate caller-side read beforehand — closing the race outright rather than merely narrowing its window. See docs/storage.md's "A dead-letter hook's write can race a late success too" for the mechanism and its empirical verification.

Developer validation: offline fixtures + manual live-test procedure (`lode-gpzn.6`)¶

Spec 10's "Live Testing" section is explicit that no live testing of the Atlassian connectors is automatable — there is no way to hit a real JIRA/Confluence Cloud tenant from the offline test gate. Validation is therefore two-tiered: an offline fixture matrix that runs on every nox -s tests, plus a documented manual procedure a developer or agent runs by hand against one real issue/page.

Offline fixtures (already in the nox -s tests gate — no new fixtures needed for this ticket). tests/test_jira_fetch.py and tests/test_confluence.py inject canned JSON through the injectable Fetcher seam (_QueueFetcher / _StubFetcher — the same offline pattern tests/test_webfetch.py uses) and already cover every outcome the spec's "Live Testing" ask names:

Connector	ok	401	404	429	paginated comments
JIRA (`tests/test_jira_fetch.py`)	`test_ok_issue_maps_to_nonempty_structured_body_including_paginated_comments`	`test_permanent_http_failure_on_issue_yields_tombstone[401]`	`test_permanent_http_failure_on_issue_yields_tombstone[404]`	`TestJiraHttpFetcher::test_transient_status_codes_raise_via_shared_classifier[429]`	same OK test — a 2-comment-page fixture, second page's text asserted present
Confluence (`tests/test_confluence.py`)	`test_ok_page_returns_clean_text_and_raw_json_payload`	`test_auth_and_404_failures_yield_tombstone[401]`	`test_auth_and_404_failures_yield_tombstone[404]`	`TestHttpxConfluenceFetcher::test_transient_status_codes_raise[429]`	n/a — Confluence maps page body only, deliberately (`lode.confluence`'s "Explicitly out of scope for this ticket")

This table exists to make that coverage explicit and auditable against the spec's own wording; the tests themselves are unchanged.

Manual live-test procedure. The fixtures above prove the fetch/classify/extract logic; they cannot prove the real Cloud REST APIs actually respond the way the fixtures assume. Run this by hand whenever validating a change that touches either connector.

Prerequisites - A JIRA Cloud and/or Confluence Cloud account, plus an API token (https://id.atlassian.com/manage-profile/security/api-tokens) and the account email used to create it. - A real, readable JIRA issue key (PROJ-123) and/or a Confluence page's full, id-bearing URL (https://<site>.atlassian.net/wiki/spaces/<SPACE>/pages/<id>/<slug> — a tiny-link or /display/... URL will not route through this connector, by design; see Draw-down rules above). - Confluence only: lode-mfts ("Wire JIRA + Confluence fetch units into refresh_external's dispatcher") must be landed on trunk. JIRA's dispatch leg has been wired in since lode-gpzn.3; Confluence's fetch unit (lode-gpzn.4) shipped standalone and offline-tested, deliberately not wired into refresh_external yet — to avoid two in-flight tickets racing on the same dispatcher lines (see lode.confluence's module docstring, "Explicitly out of scope for this ticket"). Before lode-mfts lands, a Confluence refresh job fails immediately with RuntimeError: refresh_external: no fetch unit yet for source_type='confluence' ...; bd show lode-mfts shows whether it has landed.

Fast first step: lode verify. Before running the full add/work/show cycle below, lode verify --jira / lode verify --confluence (lode-04lz) is a read-only preflight that confirms the same things steps 1-3 below check by hand — the flag is on, credentials resolve (and from which source: env var vs config.toml; the token is never echoed), and the base URL is configured or inferable — and then makes ONE authenticated, read-only GET to the connector's current-user endpoint ({base}/rest/api/3/myself for JIRA, {base}/wiki/rest/api/user/current for Confluence) to prove the credentials actually reach the tenant. It writes nothing (no DB row, no job, no embedding), so a config/auth mistake surfaces in seconds instead of after a lode add + lode work round trip. Pass a real issue key/URL (JIRA) or page id/URL (Confluence) as the optional positional argument to also dry-run real content access via the same fetch unit steps 4-6 below exercise:

lode verify --jira PROJ-123
lode verify --confluence https://<site>.atlassian.net/wiki/spaces/ENG/pages/123456/Runbook

Exit code 0 means verified reachable; any non-zero exit prints which piece is missing or wrong (inactive flag/credentials, unresolved base URL, 401/403 credentials, 404 base URL, or a transient "unreachable right now") — fix that before proceeding to the steps below, which additionally prove the full save-path wiring (refresh job, snapshot, embedding) that verify deliberately does not touch.

Steps

Set the token env vars (env vars are the primary credential source — a real token should never live in config.toml): export LODE_JIRA_TOKEN=<api-token> export LODE_JIRA_EMAIL=<account-email> # and/or export LODE_CONFLUENCE_TOKEN=<api-token> export LODE_CONFLUENCE_EMAIL=<account-email>
Flag the product(s) on in $LODE_HOME/config.toml (default ~/.lode/config.toml; create the file if it doesn't exist yet): toml jira_enabled = true confluence_enabled = true Leave jira_base_url / confluence_base_url empty to infer the API base from the pasted link's *.atlassian.net host; set one only if your Cloud site's API host differs from the link itself.
Confirm the flags took (lode config never prints a secret, but does print these two): lode config
Paste a real link into a new note — the JIRA permalink shape is /browse/{KEY}, the Confluence shape is /wiki/spaces/{SPACE}/pages/{id}/...: lode add "Debugging https://<site>.atlassian.net/browse/PROJ-123" lode add "Runbook: https://<site>.atlassian.net/wiki/spaces/ENG/pages/123456/Runbook"
Run a work pass to drain the enqueued refresh job(s): lode work
Inspect the result: lode show <note-id> lode dump-html <note-id>

Exact expected outputs

Step 4: each lode add prints a note id and exits 0.
Step 5: lode work's output includes one outcome line per link, exactly refreshed PROJ-123: ok / refreshed 123456: ok (the literal string lode.drawdown.refresh_external returns on success), followed by drained N job(s) with N >= 1. No RuntimeError should appear: no api_base persisted for external_id=... means the flag/token wasn't resolved yet at lode add time (redo from step 1 and re-paste); no fetch unit yet for source_type='confluence' means lode-mfts hasn't landed (see Prerequisites).
Step 6, lode show <note-id>: the note body, then edges: listing -> PROJ-123 (or -> 123456) with a nested line jira · snapshot <id> · as of <timestamp> [un-refreshed] (or confluence · ...) — un-refreshed is the fresh, non-tombstoned, non-withheld state (lode.enrichment_view's naming — not a warning). embedded: yes (N passage(s)), N >= 1, once the snapshot's own embed job has also drained (a separate queued job from refresh — re-run lode work once more if it still reads embedded: no).
Step 6, lode dump-html <note-id>: prints the snapshot's raw JSON payload verbatim — for JIRA, valid JSON with an "issue" key (fields.summary, renderedFields.description) and a "comments" key (a list); for Confluence, valid JSON with "id" matching the pasted page id and a body.view.value string holding the page's real rendered HTML.
Optional negative check: a deliberately wrong token reproduces the offline 401 fixture case — lode work still reports refreshed ...: tombstone (http_401) (never a crash; the classified reason is appended in parens per lode-gpzn.5 -- work-pass error visibility -- never the token or response body, only the fetch unit's own short machine tag), and lode show <note-id>'s external line reads [stale] instead of [un-refreshed] (lode.enrichment_view: stale is the state name for a tombstoned head snapshot).

Atlassian connectors (JIRA + Confluence Cloud, `lode-gpzn`)¶

A second and third connector alongside the generic web path above: JIRA and Confluence Cloud links draw down via authenticated REST APIs into structured snapshots, instead of trafilatura-scraping an auth-fronted login page (what the web path alone would do against either product today). Data Center / Server is explicitly out of scope, deferred — see decisions.md. Each product is its own feature flag, default off: jira_enabled / confluence_enabled (configuration.md has the full knob table — base-URL overrides, the env-var-primary/config.toml-fallback token resolution, the secret=True exclusion from lode config — not repeated here). A connector is active only when its flag is on and Basic-auth credentials (account email + API token) resolve (lode.config.jira_active / confluence_active); flag-off or unresolved credentials means a matching link falls straight through to the unchanged generic web path (login page ⇒ tombstone) — never an error, always a quiet degrade. Credentials resolve env-var primary (LODE_JIRA_TOKEN/LODE_JIRA_EMAIL, LODE_CONFLUENCE_TOKEN/LODE_CONFLUENCE_EMAIL), with an optional config.toml fallback — no secret is required to live on disk, but one may, in plaintext; there is no OS-keyring integration (decisions.md records this as a deliberate deferral, not an oversight).

Raw httpx2, no Atlassian SDK. Both lode.jira_fetch.JiraHttpFetcher and lode.confluence.HttpxConfluenceFetcher are single-purpose, hand-rolled clients that implement the exact same lode.webfetch.Fetcher protocol (fetch(url: str) -> RawResponse) the web connector's HttpxFetcher already does — production wires the real client, tests inject a stub, so the offline gate never makes a network call. No SDK dependency, no parallel fetch abstraction; this is the same seam applied to two more connectors, not a new one.

Semantic `external_id`, not a URL (locked decision 3 + refinement A)¶

Unlike a web external_id (a canonical URL string), an Atlassian external_id is the semantic identity the pasted link encodes — a JIRA issue key (ABC-123) or a Confluence page id — parsed out by lode.drawdown._classify_atlassian at link-detection time, synchronously, with no network I/O (owner decision F: this step must never block the note-save transaction on an auth round-trip). The payoff: a browser permalink, an API URL, and (where the id is present) any other URL form of the same issue/page all parse to the same key and dedup onto one externals row — the identical "one canonical node per identity" guarantee External identity above states for the web connector, now true across URL shape too, not just across repeated pastes of one exact URL.

Consequence — the persisted API-base seam (decision A). Because external_id is no longer itself a fetchable URL, the async refresh job handler can't rebuild a request from external_id alone the way the web path does. The inferred-or-configured API base (jira_base_url/confluence_base_url when set; otherwise the pasted URL's own *.atlassian.net scheme+host, inferred by lode.drawdown._resolve_api_base) is therefore persisted synchronously at detection time, on a new api_base column on the externals row (src/lode/schema.sql) — the same transaction that writes source_type and external_id. lode.jira_fetch.fetch_jira_issue and lode.confluence.fetch_confluence_page both take external_id + api_base as explicit parameters and rebuild the request URL as {api_base}+{external_id} (JIRA: {api_base}/rest/api/3/issue/{key}; Confluence: {api_base}/wiki/rest/api/content/{page_id}), reading api_base off the row (lode.drawdown's shared _refresh_atlassian leg of refresh_external's dispatcher, parameterized on the fetch callable and SOURCE_TYPE_* — lode-40zj deduped what were originally two byte-identical _refresh_jira/_refresh_confluence functions). This is a general seam, not Atlassian-specific plumbing: any future connector whose semantic id isn't itself a URL can reuse the same "persist the base at detection, rebuild at fetch time" shape.

Bare JIRA issue keys (lode-2o45)¶

A JIRA issue only drew down when the note carried the full permalink (https://acme.atlassian.net/browse/PROJ-123) — a bare PROJ-123 in prose never entered the pipeline. jira_projects (configuration.md, runtime, list[str], default []) turns that on, but only for an explicitly configured project-key allowlist — never a generic [A-Z]+-\d+ scan, which would false-positive on UTF-8, SHA-256, COVID-19, ISO-8601, and similar hyphenated tokens that are common in notes but are not issue keys.

Allowlist-not-pattern, and case-sensitive (owner decisions, 2026-09-08). ["PROJ"] matches PROJ-1234 but not PROD-1234 or PROJECT-1 — the same whole-key-boundary rule the no-egress-scope source_type="jira" match already uses. The match is case-sensitive: proj-123 in prose does not match a configured PROJ — tickets should visually stand out in a note, and lowering the false-positive surface matters more here than symmetry with the URL path (which persists whatever case the pasted URL used). The key is persisted exactly as matched (case preserved), so a bare PROJ-42 and a pasted /browse/PROJ-42 dedup onto the same externals row — one row, one edge per note, one job, exactly like two equivalent URL forms.

Activation: an add-on that cannot fire until the connector itself is active. The bare-key scan requires all of: jira_active(settings) (flag on + credentials resolve), jira_projects non-empty, and jira_base_url set. jira_base_url is mandatory here — unlike the URL path, where it's an optional override — because a bare key carries no host of its own to infer an API base from. Its absence is not a Settings() construction error: the connector's URL-based routing keeps working exactly as today, and the bare-key scan is simply inert. That misconfiguration (connector active + jira_projects set + jira_base_url empty) is surfaced through lode verify --jira and lode status's "Action needed" line — never as a per-save warning or a crash. With any gate false, a bare key is inert text: there is no generic-web fallback for it (no URL to scrape), so silence is the correct degradation.

Synchronous and network-free, same as the URL path (owner decision F, above) — a regex over the body with the configured prefixes alternated, word-boundary anchored on both sides (src/lode/drawdown.py::iter_bare_jira_key_spans). A key inside a pasted URL's own span is not double-detected: the URL already routes through _classify_atlassian, and detect_and_enqueue_drawdown skips a bare-key match whose span overlaps any URL span.

Backfill (in scope). lode.jira_backfill._jira_backfill reclassifies every existing explicit edge under current routing (unchanged for URLs), and separately scans every live note head's body directly for a bare key that predates jira_projects listing its prefix — such a key was never linked at save time, so there is no existing edge to reclassify. Both legs share the identical detector the save path uses, and both are idempotent: a key already linked from a given note is a no-op on the next backfill pass.

TUI link-open: built (lode-dube). A bare key under the cursor opens as {jira_base_url}/browse/{KEY} on Ctrl+N, from every screen that binds it, via the same iter_bare_jira_key_spans matcher the save path uses — so "what opens == what draws down" holds for bare keys too, not just URLs. src/lode/tui/screens/_link_open.py stays pure: its functions take jira_projects/jira_base_url as plain values. The activation gate is drawdown._bare_jira_scan_active itself, applied in exactly one place (bare_jira_open_args, called by the module's glue function), never a second copy of it. A cursor inside a pasted URL still yields the URL — the URL spans are matched first — so the two shapes never contend.

Confluence: only an id-bearing URL routes (decision F)¶

Only /wiki/spaces/{SPACE}/pages/{id}/... — Confluence Cloud's page-id-bearing URL shape — routes to the connector (lode.drawdown._CONFLUENCE_PAGE_RE). A tiny-link (/wiki/x/AbCdE) or a legacy display URL (/display/{SPACE}/{Title}) carries no page id in the URL itself, and resolving one to an id would need an API round-trip — exactly the network I/O link-detection must not do (same owner-decision-F reasoning as the API-base seam above: detection stays synchronous). Both forms fall through to the generic web path today (login page ⇒ tombstone), on a matched Confluence host, same as flag-off — this is a deliberate scope boundary, not a routing bug, and is tracked as a known, unfixed gap in decisions.md rather than silently accepted. JIRA has no analogous gap: its only Cloud permalink shape, /browse/{KEY}, already carries the issue key directly (lode.drawdown._JIRA_ISSUE_RE); anything else on a matched JIRA host (a board, a dashboard, a search) has no semantic id either and falls through the same way.

Body representation: rendered HTML, the existing extractor (decision E)¶

Neither connector writes a bespoke parser for its product's native body format — JIRA REST v3's Atlassian Document Format (ADF, a nested JSON doc) or Confluence's storage-format XHTML (full of ac:structured-macro/ri:... macro elements). Both instead request the product's own server-side rendered HTML — JIRA: expand=renderedFields on the issue plus expand=renderedBody on each paginated comment (lode.jira_fetch._fetch_comments loops /rest/api/3/issue/{key}/comment?startAt=… until every comment is collected); Confluence: expand=body.view — and run that HTML through the existing readability extractor, lode.webfetch._extract (trafilatura), the same one the web connector already uses. No connector-specific extraction code exists for either product. The full raw JSON response is kept verbatim as raw_payload either way (via FetchResult.raw_html, the same field name the web leg uses for provenance, holding JSON here instead of HTML) — for anyone who later wants the ADF or the storage-format markup, it isn't discarded, just not what gets extracted.

Shared classification, shared taxonomy (decision C, `lode-gpzn.13`)¶

Both connectors classify every HTTP outcome through the same connector-neutral lode.fetch_outcome.classify_http_status the web path uses — see Fetch-outcome taxonomy above for the full OK / TOMBSTONE / TRANSIENT mapping and its worker-queue consequences (401/403/404 ⇒ tombstone; 408/429/5xx/network/timeout ⇒ raise into the existing attempts/backoff/dead-letter machinery). No connector reimplements the status-code mapping locally; lode.worker._refresh_dead_letter_hook is generic over source_type for the same reason (reuses whatever source_type the externals row already carries rather than assuming web). This was a prerequisite refactor (lode-gpzn.13, extracted out of lode.webfetch behavior-preservingly) that both lode-gpzn.3 (JIRA) and lode-gpzn.4 (Confluence) built on rather than each carrying their own copy of the 401/403/404-vs-408/429/5xx split.

Refresh policy and backfill are reused unchanged¶

Neither connector introduces a second staleness/scheduling mechanism or a second re-draw-down command. Refresh policy's TTL sweep and Backfill's framework both operate on externals rows generically, keyed on source_type/external_id/api_base — a JIRA or Confluence row is just another row to either mechanism, needing no per-connector branch in either. The Backfill section above already documents the one connector-specific nuance that follows from decision D — a fresh migration mints a never-tombstoned semantic external and enqueues a plain refresh; the tombstone-exclusion override matters only on an idempotent re-run — so it isn't restated here.

Link-rot immunity (the payoff that justifies draw-down)¶

Because we snapshot externals instead of storing bare URLs, the knowledge graph is immune to link rot: when the ticket is deleted, the wiki reorganised, the page taken down, the mirrored snapshot — and everything the AI derived from it — survives. The opposite of bookmarks. Principle: always snapshot, never store a bare URL.

Privacy (consequence of aggregation)¶

Single-user does not mean low-stakes. Once this box holds email + internal tickets + repo contents, it is a concentrated high-value target — and the precise privacy claim matters.

What leaves the box, and what doesn't¶

Content never leaves the box for indexing. Enrichment and Q&A are explicit, governed egress.

Not "content never leaves the machine" — that's false and the headline must not say it:

Local, never leaves: chunking, embeddings, reranking, NLI entailment. Indexing and retrieval are fully on-box.
Leaves the box to the configured cloud LLM provider — provider-selected via llm_provider, Anthropic by default: enrichment (default: Haiku, every note, background) and Q&A (default: Sonnet/Opus, the retrieved passages — which can include mirrored ticket/email/repo snapshots — per question). Setting llm_provider = "openai" sends the identical payloads to OpenAI/Azure instead — the privacy boundary is leaves the box to whichever provider is configured, not "leaves the box to Anthropic" specifically. The aggregation that makes this box valuable is exactly what Q&A ships into the cloud prompt, often invisibly, regardless of which provider is active.

Two redactions, aimed at the right legs¶

Redaction is not one control. Because embedding is local, redacting before it only affects local retrievability — it does nothing about egress, since the secret still sits in versions.body and is still sent to the configured cloud LLM at enrichment/Q&A time:

Redact-before-index — a pasted .env / API key doesn't become locally retrievable (vector
FTS). Local-at-rest concern.
Redact-before-egress — strip known secret patterns from the enrichment payload and the Q&A context before they're sent to the configured cloud LLM. This is the control that actually limits cloud exposure, and it's the one §6 originally omitted.
purge (the corrective half) remains the only thing that removes the durable copy from versions.body.

No-egress tier (for genuinely sensitive notes/sources)¶

A note, or a specific already-captured external source, can be marked no_egress:

still captured, chunked, embedded, and locally retrievable (keyword + vector);
never sent to the configured cloud LLM — no enrichment, and excluded from cloud Q&A context;
in an answer it is cited as "present, withheld from cloud synthesis" rather than silently dropped, so the user knows relevant material exists but was kept local. (A local-LLM fallback that could synthesize over withheld notes is a future option — see decisions.md.)

This keeps work secrets in the KB and retrievable while guaranteeing they never reach the cloud.

The control surface for one already-captured external is lode no-egress <external_id> (--clear to undo it), which flips externals.no_egress; every send path (enrichment, Q&A) reads the flag off the row, so setting it is the only step needed for a row that already exists (lode-w0h.7). This is a resource-level control — one external_id, one row. Marking a whole source — a repo, a ticket project, a host — broader than any single resource is a separate mechanism, scope rules, below.

`settings or Settings()` fallback survey (decided, `lode-xa5d`)¶

Settings.no_egress_default is the seed for no_egress on every newly-inserted notes/externals row (root note create, externals.ingest_snapshot, backfill.mint_external, drawdown.detect_and_enqueue_drawdown — lode-a43n, lode-ge8w). Every one of those functions, like ~30 other functions across the codebase, declares settings: Settings | None = None and falls back with the house pattern settings = settings or Settings() when the caller omits it — but a bare Settings() construction is plain-defaults, not the user's config.toml (that file is read only by load_settings(), called once per CLI/TUI entry point and threaded down — lode.config.load_settings's own docstring). Falling back silently on one of the functions above would re-introduce, for a future forgetful caller, exactly the bug lode-a43n and lode-ge8w were filed to fix: a new row seeded no_egress=0 regardless of what the user configured.

Surveyed conclusion: every live production caller of the five functions above already threads a real, resolved Settings instance (confirmed by walking the call graph — lode.cli's entry points, lode.repository.Repository, lode.tui.services.*); the only omissions found were in tests. Because ~30 other settings or Settings() sites are not privacy-bearing (they don't decide a no_egress seed — e.g. drawdown.canonicalize_url, redact.py, retrieval/embedding tuning knobs) and a bare Settings() there really is "library default is fine, this is a convenience for a standalone caller," tightening only the five privacy-bearing sites — rather than making settings required everywhere — was the chosen fix, to avoid churning ~20 unrelated call sites (many exercised across a dozen test files) for no safety gain.

The fix: those five sites (lode.versions.save, lode.externals.ingest_snapshot, lode.backfill.mint_external, lode.drawdown.detect_and_enqueue_drawdown, lode.repository.Repository.save) fall back via lode.config.default_settings_for_missing_arg(caller) instead of a bare Settings() — same defaults-only Settings() underneath, but it logs at WARNING first, so an omitted settings= at one of them is loud (a log line an operator or CI can notice) rather than a silent privacy regression with no test failure and no error. settings: Settings | None = None was kept (not made required) at all of them, since making it required would only shift the failure from "loud log line" to "TypeError," at the cost of forcing every test caller of these widely-used functions to start passing settings= explicitly for no additional safety — the loud fallback already makes an omission visible. Every other settings or Settings() site is unaffected and stays the plain house pattern documented on load_settings.

Masking wrappers count as privacy-bearing too. A caller one frame above a hardened site that keeps the plain settings or Settings() fallback resolves the omission itself and hands the write site a non-None, defaults-only Settings — so the WARNING can never fire for that path and the omission is silent again, just one frame higher. Two such entry points therefore use the loud fallback as well: lode.tui.services.capture.save_capture (mints a fresh note, so it seeds notes.no_egress) and lode.tools.fetch_for_ask (mints a fresh external via ingest_snapshot, and separately consults settings for the no_egress_denied egress gate). The line is drawn at wrappers whose own job is to mint the row; the other Repository.save wrappers (tui.services.edit.save_edit/delete_note, tui.services.reconcile.reapply) only ever update an existing note, and root-create is the only path that seeds no_egress (lode.versions._save_core), so they stay on the house pattern. lode.worker.drain also stays on it deliberately: it is a general executor whose handlers may write rows, not a mint site. Note the carve-out rests on that alone, not on warning volume: lode.cli.work -- drain's only production caller -- threads a real _resolve_settings() result, so a hardened drain would in practice fire the privacy warning on no invocation at all.

No-egress scope rules (decided, `lode-35nu.11.8`)¶

The per-row flag above cannot cover an external that has no row yet — exactly the resources a fetch/search tool exists to reach (lode-35nu.11.1/.2). no_egress_scopes (configuration.md) is a declarative, config-held rule set — JIRA project key or URL host — evaluated live against a candidate (external_id, source_type) at decision time, with no row required and no write to externals ever performed. Rules compose with the per-row flag (either denying is a denial); an already-captured external newly covered by a rule is withheld starting at its next send, with no migration or backfill.

Confluence space-key scoping is out of scope (human decision). The obvious rule shape for Confluence would be a space key, matched the same way a JIRA project key is matched against an issue key. It cannot be built without a schema change: drawdown.py's _CONFLUENCE_PAGE_RE persists only the numeric page id into externals.external_id, deliberately discarding the space key at detection time (see that regex's own comment) — confirmed by grepping every Confluence code path (drawdown.py, confluence.py, confluence_backfill.py): the space key is never persisted anywhere, on the externals row or otherwise. A space-scoped rule therefore has no space information to ever match against, for a captured or not-yet-seen Confluence external alike — a structural limitation of the current data model, not a missing wiring pass. Closing it needs either a space_key column on externals (a schema change out of scope here, and not depended on) or a richer predicate signature a future tool call site could supply space info through (considered and explicitly declined — see lode-35nu.11.8's notes; the general mechanism is not built speculatively). Because a confluence scope rule can never match anything, Settings rejects one at config-load time with a clear error rather than accepting it as a silent no-op — see the field validator on Settings.no_egress_scopes. JIRA project-key and URL-host scoping are unaffected and fully supported today.

Tool calls are egress too (decided, `lode-35nu.11.5`)¶

Tool-augmented Ask opens a send path the two redactions above do not cover. gate_qa_egress (lode.egress) guards the synthesis send — the passages going to the model. It does not guard what the model then puts into a tool call: a search string or an issue key composed by the model and shipped to a third-party API is content leaving the box through a path with no gate and no egress-log row.

First, the correction to the threat as originally filed. The motivating worry was "the model composes a search string out of a no_egress note's content." That specific leak is not reachable today, and the doc should not imply it is: no_egress material is excluded from cloud Q&A context (above), so the cloud model never receives it and cannot compose anything from it. Banning tools outright whenever no_egress material is in the corpus — the strictest option considered — was therefore rejected as guarding a path the architecture already closes, at the cost of disabling tools in exactly the sessions where the user has sensitive notes, which is most of them.

What is real, and what the decided mechanism actually covers:

The audit gap stands on its own, whoever composed the string. A tool call ships bytes to a third-party service. storage.md's one-audit-row-per-egress rule applies, and today no row is written. Every tool call writes an egress_log row — purpose = 'tool', the tool invoked, the destination external_id/service, the arguments as sent, and any redactions applied. This is required independent of any threat model; it is what makes "what of mine has gone to the cloud, and when?" remain a true claim once tools exist.
The user's own question text is unredacted and reaches both the model and any tool argument derived from it. That path predates tools — it is Ask's existing surface — but a tool call forwards it to a second party. So gate_qa_egress's redaction runs over tool arguments, not just over passages. Reusing the existing gate rather than writing a second one is the point: the secret patterns it strips are the same patterns, and a divergent second redactor would be a correctness hazard the moment either side is tuned.
A tool call whose destination source is itself no_egress is forbidden. The flag already means "this source's content never goes to the cloud"; querying it live and feeding the result to cloud synthesis would launder exactly what the flag forbids.

Why redaction is specified even though the composed-from-no_egress leak is unreachable: a local-LLM fallback synthesizing over withheld notes is an explicitly-live future option, and under it the model would see no_egress content and could put it in a tool argument. Choosing the mechanism now, while it costs one reused function call, is cheaper than retrofitting it onto a path that by then has users.

Web-fetch destination guard: SSRF via a model-chosen URL (decided, `lode-xwah`)¶

The model-chosen web_fetch destination on the ask path (lode.tools.fetch_for_ask) is adversary-influenced in the same sense the tool arguments above are: whatever the model puts in the URL, this box issues a real HTTP request to. Beyond no_egress (which withholds a known sensitive source, not an arbitrary address) and beyond is_no_egress_scoped (a config-declared allow/deny list), nothing on trunk validated that the destination itself isn't an internal address until this ticket — http://169.254.169.254/latest/meta-data/-style cloud-metadata reads and http://127.0.0.1:<port>/ reads of same-box services are both otherwise reachable.

lode.webfetch.GuardedHttpxFetcher (an HttpxFetcher subclass, constructed only by lode.tools._fetch_web and injected via the fetcher= seam fetch_for_ask already threads — never the draw-down path, never JIRA/Confluence) closes two gaps a single pre-fetch address check cannot:

Redirect chains. httpx2's own follower has no per-hop hook, so an allowed public host's 3xx response could still point the next request at an internal address without ever being inspected. GuardedHttpxFetcher follows redirects manually (follow_redirects=False on every per-hop client) and validates every hop's destination — the original URL, and every redirect Location — before that hop's request is issued, not merely before its response is trusted.
DNS rebinding / TOCTOU. The pre-hop check resolves the host itself; httpx2's transport resolves the same host again, separately, for the real connection. A hostile short-TTL resolver can answer the two differently and defeat a pre-check-only guard outright — the cheaper of the two attacks for an adversary who already controls the domain and its TTL, not a corner case. GuardedHttpxFetcher additionally validates the address httpx2 actually connected to (response.extensions['network_stream'].get_extra_info('server_addr')) after every hop, before that hop's response is used for anything (followed as a redirect, or returned as the final result) — so a rebind that fools the pre-check is still caught.

An address is disallowed if it is private, loopback, link-local, reserved, multicast, or unspecified (ipaddress.IPv4Address/IPv6Address's own classification), plus two ranges no ipaddress attribute flags and which are therefore live bypasses of the attribute check alone: 100.64.0.0/10 (RFC 6598 carrier-grade NAT — routinely a container or ISP-internal network) and fec0::/10 (RFC 3879 deprecated IPv6 site-local, which ipaddress even reports as is_global). IPv4-mapped IPv6 addresses (::ffff:127.0.0.1) are unwrapped and judged as the IPv4 address they really are. The scheme is allowlisted to http/https. A host that fails to resolve at all, and a peer whose address cannot be read at all, are refused rather than passed through unclassified — fail-closed, the same posture is_no_egress_scoped takes. Fail-closed on the peer is only correct because the check runs on a still-streaming response: httpcore can only answer get_extra_info('server_addr') while the connection is live, so after a non-streaming read a Connection: close server's socket is already gone and the call raises OSError.

Relationship to lode.tools._refuse_private_web_destination. A sibling ticket (lode-ejfv) was, at the time this ticket was scoped, adding a private-address guard directly in lode.tools (checked on the initial URL and again on the post-redirect final URL) — but had not yet landed on trunk when this ticket was built, so trunk carries no such function to remove or extend. GuardedHttpxFetcher was built standalone against trunk as it stood, per this ticket's own SHAPE decision to guard at the fetcher layer rather than extend a lode.tools-level check either way. Land-time note: if lode-ejfv's _refuse_private_web_destination lands first or alongside this branch, its guard is strictly coarser on the axes this ticket was scoped for (initial-URL + final-URL only, no per-hop and no rebinding check) — but it is not simply redundant, and "delete it" is not automatically the right reconciliation. It covers one case GuardedHttpxFetcher structurally cannot: _fetch_web installs the guarded fetcher only when the caller injects no fetcher, so any caller that injects one gets no address policy at all, whereas the lode.tools-level check runs regardless. Whoever lands both decides: delete it, or keep it as the injection-proof outer check. See docs/decisions.md for the open reconciliation item.

Web-fetch destination guard (decided, `lode-ejfv`)¶

Discovered while reviewing tool-augmented Ask (lode-35nu.11.2): before that ticket, every URL lode fetched originated from something the user wrote (a link pasted into a note, drawn down by lode.drawdown). With web_fetch exposed to the Q&A loop, the destination is chosen by an LLM whose context can include attacker-influenced content — a fetched page, a JIRA ticket body. Two consequences: (1) the destination is otherwise unconstrained — no_egress_scopes is a deny list, so it only covers hosts the user thought to name, and a model steered by fetched content could be pointed at an internal address (169.254.169.254, localhost, an intranet host) from the user's own network position; (2) there is an exfiltration shape (content encoded into a chosen URL), not just a read — already audited via the purpose='tool' egress-log row (detection, not prevention).

Decided: a private/loopback/link-local/reserved/multicast address guard on the web_fetch destination, scoped to the ask path's SOURCE_TYPE_WEB leg only (lode.tools._refuse_private_web_destination). The initial destination's scheme is restricted to http/https and its resolved IP address(es) are checked against those ranges — refused, with nothing fetched and no externals row minted, before any network call, mirroring the no_egress check's ordering and rationale immediately above. JIRA/Confluence fetches are not in scope: their destination is a configured api_base, not a model-chosen host.

Alternatives considered and rejected:

An allowlist of hosts/schemes for web_fetch. Rejected as the wrong shape for a general-purpose research tool — it would need constant maintenance as the user's browsing needs change, and does nothing an address-range guard doesn't already cover for the concrete threat (an internal address, not an untrusted-but-public one).
Requiring a web_fetch destination to already have an externals row (tools may only re-read what the corpus already knows; discovery stays JIRA/Confluence-only). Rejected — it removes the entire point of exposing web_fetch to Ask (discovering and citing a new web source mid-question), for a threat the address-range guard already closes more narrowly.
Accepting the risk, with Settings.tools_enabled=False-by-default as the whole mitigation. Rejected as too weak given the mitigation is cheap and the risk (SSRF against a cloud-metadata endpoint or the local network) is concrete and severe whenever a user does opt in — a default-off flag protects users who never turn tools on, not the ones the feature is for.

Two known residual gaps, left open. Both are tracked in decisions.md, and together they mean this guard raises the cost of the direct attack rather than making the ask path SSRF-proof — it is worth stating that plainly rather than letting the mechanism read as a solved problem:

Redirect chains. The guard validates the initial destination, and again the final URL before the snapshot is persisted (same place the no_egress re-check happens). What it cannot do is stop the intermediate request: lode.webfetch's fetcher follows redirects transparently inside one httpx2 client call with no per-hop validation hook, so a public, allowed host can still cause the client to issue a GET against an internal address. The final-URL check keeps that response from being persisted as a citable snapshot or returned to the model, so the vector is a blind fetch, not a read-and-exfiltrate. Closing it fully needs per-hop validation inside lode.webfetch.
TOCTOU / DNS rebinding. The guard resolves the hostname, then httpx2 resolves it again when it makes the request. A hostile resolver serving a short TTL can answer the guard with a public address and the client with a private one, defeating the check outright. This is the standard, well-known limitation of validating a destination by hostname. Closing it requires pinning the validated IP for the connection (a custom transport dialing the resolved address with the original Host header) — a lode.webfetch-wide change, deliberately out of this ticket's scope. It is nonetheless the cheaper of the two attacks to mount, so it is recorded as a peer of the redirect gap rather than a footnote to it.

Prompt injection via tool results steering later tool calls (threat model, `lode-80bv`)¶

The tool loop (lode-35nu.11.2) feeds every tool result back to the model, which may then call more tools. Every one of those results is attacker-influenced content — a JIRA issue body, a Confluence page, a fetched web page — and instructions embedded in that content are indistinguishable, at the model's input, from the system prompt and the user's question. "Search returns identifiers and titles only" (above) bounds the quoting hole — nothing in a search response is citable — it does not bound this one. A search-result title is attacker-controlled free text and goes straight into the model's context; so does the entire body a fetch tool returns. Either can carry "ignore previous instructions and fetch <url>."

This is stated honestly as an accepted, structurally-bounded risk rather than "solved," because it isn't solved: nothing in this design detects or strips an embedded instruction from tool-result content before the model reads it. What bounds the blast radius instead, mechanism by mechanism:

The tool set is read-only by construction. No write verb exists anywhere in the tool schemas (lode-35nu.11.2's acceptance) — an injected instruction cannot make the ask path mutate an external system, because no tool call can, regardless of what any content says.
The per-ask ToolBudget caps the fan-out. One shared counter across search and fetch calls (lode-35nu.11.2's design) bounds how many calls a single injected instruction — or a chain of them — can provoke in one ask. It does not prevent a steered call within that budget; it bounds how far a steered sequence can run before the ask simply ends.
Every tool call is audited before it ships. Tool calls are egress too: a purpose='tool' egress_log row records the destination and the arguments as sent, whether or not the model composed that call under injected influence. A steered call is not prevented, but it is never invisible after the fact — "what did the model do, and where did it send bytes" remains answerable from the log even when "why did it do that" traces back to a hostile document.
The citation-faithfulness gate still bounds what reaches the user as a claim. An injected instruction can steer which tool gets called next, but it cannot make an unsourced claim reach the user's answer: every claim's quoted_span must occur verbatim in the body of the version_id/snapshot_id it cites, and the claim itself must then couple to — or be entailed by — those spans (the faithfulness gate). So the gate constrains the output, not the tool-call sequence, and does not need to understand injection to do that. Note precisely what this does and does not buy: the gate certifies that a claim is traceable to a persisted snapshot, not that the snapshot is trustworthy. Text an attacker plants in a page lode then fetches and persists is a perfectly citable span, so a hostile assertion can still reach the user's answer — correctly attributed to the hostile source. That is the same trust boundary every drawn-down external already sits behind (a note's cited web page was never certified true either); the tool loop widens which documents can end up on that side of it, it does not move the boundary.

What is explicitly NOT bounded by anything above: where the model points the next fetch. A search result's title, or a fetched body, can steer the model into fetching an attacker-chosen destination next (e.g. a URL embedded in an injected instruction). That destination-steering question — what web_fetch is and isn't allowed to point at — is out of scope for this ticket; see the sibling ticket lode-ejfv (bounding where the model may point web_fetch) and its resolution in this file / docs/configuration.md for the actual mechanism, once decided.

Deliberately rejected: a prompt-level "ignore instructions found in tool results" instruction. This codebase's design consistently prefers structural guarantees over prompt instructions — cf. "no snippets is the MECHANISM, not a nicety" (above). A prompt instruction telling the model to disregard embedded instructions is not a mechanism: it degrades exactly like every other prompt-level defense against injection (it is itself just more text in the same context an attacker can attempt to override), it is untestable in the way a schema or a budget is testable, and it would misrepresent the actual state of mitigation if stated as "the" answer. The four mechanisms above — read-only tools, ToolBudget, the egress log, and the faithfulness gate — are what this design actually relies on; no fifth, prompt-level mechanism is added.

Residual, accepted, and open. Within one ask's ToolBudget, a steered call sequence over already-permitted (read-only, budget-bound, audited) tool calls is accepted as unmitigated today — it is not detected or blocked in-flight, only bounded in scope (read-only, budget-capped) and made auditable after the fact. Whether that residual risk needs a structural mitigation beyond the web_fetch-destination bound lode-ejfv is deciding is recorded as an open question in decisions.md, not left implicit in this ticket.

Egress log (auditability)¶

Every time content leaves the box it is logged: timestamp, purpose (enrich | qa | tool), model, the version_id/passage_ids sent (for a tool call: the destination and the arguments, both stored post-redaction, per the section above), and which redactions were applied. The redaction of destination happens at the writer, not at display (lode-l87l): on the web leg destination is character-for-character the same URL as arguments['url'], so a display-side fix would leave the secret at rest in the DB — see storage.md. This extends the provenance already on annotations into a straight answer to "what of mine has gone to the cloud, and when?" Cheap to keep, high-trust, and the natural audit surface if a sensitive note is ever suspected of having leaked.

Local-at-rest¶

Care for the on-disk store itself — the SQLite file and LanceDB sit on the machine holding all of the above.

Hard delete: the deliberate immutability break (corrective half)¶

Append-only + content-addressing means a pasted secret otherwise lives in versions.body forever, and a normal delete only writes a tombstone — the bytes survive. Because this box aggregates sensitive data, there must be an escape hatch that violates immutability on purpose:

purge operates at version or whole-note granularity (v1 — substring/span redaction is deferred, see decisions.md). It overwrites the body of the targeted version(s) with a redaction marker ([purged YYYY-MM-DD]) and sets purged_at. Node identity, parent_version_id, op, and created are kept, so lineage and undo structure survive — only the sensitive bytes die.
It sweeps the note's whole chain, including soft-deleted (tombstoned) notes — a secret pasted then edited-around persists in older versions.
It cascades to the cache: drop every derived entry referencing the purged versions (LanceDB vectors, FTS rows, source: ai annotations), then re-derive cheaply/locally so nothing leaks through the index. source: user annotations stay (metadata, not content).
Hash consequence (accepted): a purged body no longer hashes to its version_id; that id stays as the historical identifier, flagged purged, and is no longer recomputable. This is the cost of an explicit immutability break, taken knowingly.

lode — External sources & the knowledge graph¶

Externals fit the same model¶

The broken assumption: external staleness is NOT topological¶

External identity — same two-id split¶

A query result has no identity — discovery is not citation (decided, lode-35nu.11.5)¶

Ask-time snapshots are first-class, with a provenance marker (decided, lode-35nu.11.5)¶

URL canonicalization (decided, lode-w0h.3; userinfo stripped, lode-0as)¶