lode — Retrieval¶

The pipeline behind the primary bet (cited Q&A — design.md §2). It runs over the heterogeneous graph of owned notes + mirrored externals; the node model and the trust gradient it ends on are defined in externals.md.

Index heads only¶

For notes, index the current head version; for externals, the latest snapshot, marked with its age. Never index full history — a note edited 5× would return 5 near-duplicate hits and cite a stale copy. History exists for audit, undo, and annotation migration (storage.md) — not retrieval.

Chunking: passages are the retrieval unit¶

A whole-note embedding dilutes recall (a long meeting note mixes standup + a deploy decision + an unrelated bug into one averaged vector) and forces citations up to "this 600-word note." So the head version's body is chunked into passages, and the passage — not the note — is the unit that gets embedded, lexically indexed, fused, and reranked. Both retrieval legs must rank the same unit or RRF fuses apples and oranges, so FTS5 indexes passages too.

Structure-aware, with a token fallback. Split on the note's own structure — markdown headings, paragraphs, list items (meeting notes have sections; runbooks have numbered steps) — then sub-split any block over N tokens with slight overlap so passage size stays bounded. Deterministic and local — no LLM, no model, no network in the chunker. The passage structure (passages rows + FTS5 index) is written synchronously on save by the model-free LexicalCacheBackend (settled lode-xyb), so a just-captured note is keyword-findable immediately; only the embedding step (LanceDB vectors) is async via the worker.
Small-to-big retrieval. Match on the small passage (precision), but expand each hit to its parent block (parent_block in storage.md) so the Q&A LLM gets enough surrounding context to synthesize — while the citation still points to the precise passage/span. Match a sentence, reason over its section, cite the sentence.
Cleaner spans feed the faithfulness gate. A coherent passage is a far better unit for extractive coupling and NLI entailment (below) than an arbitrary slice — so chunking quality directly sets citation-verification accuracy.

Passages are regenerable cache, re-chunked per head version (storage.md). Chunk size N and overlap are tuning knobs, deferred to the eval harness (decisions.md).

The v1 retrieval pipeline is hybrid, fused app-side¶

retrieve(q):
  L     = FTS5.search(q)          # lexical / BM25 over passages — sync, always fresh
  V     = lancedb.search(emb(q))  # dense vector over passages — async cache
  fused = RRF(L, V)               # app-side Reciprocal Rank Fusion (~20 lines)
  top   = rerank(q, fused)        # local cross-encoder stage (toggleable)
  big   = expand_parents(top)     # small-to-big: passage hit → its parent block for context
  ctx   = graph_expand(big)       # GraphRAG: traverse edges from the seeds' notes
  return trust_rank(ctx)          # trust gradient orders the final context

Diagram

Hybrid, never vector-only. Pure dense retrieval underperforms on keyword-heavy technical queries ("rotate the staging certs"); the lexical leg carries those — and, because FTS5 is the synchronous index (design.md save path), it also covers a just-written note before its vector lands.
emb(q) is the asymmetric query side. The pinned local model (nomic-embed-text-v1.5, stack.md) is asymmetric: documents are embedded with a search_document: prefix, queries with search_query:. Embedding the question with the document prefix lands it in a subtly wrong region of the space and softens dense recall, so the embedder exposes a distinct embed_query method that applies search_query: — the one query-embedding primitive the read side (emb(q)) and the eval harness both call, paired with the document-side embed_passages.
Fusion is app-side RRF. One lexical index (FTS5, fresh) + LanceDB for vectors only. LanceDB's own native hybrid goes unused — its lexical leg would lag (it's the async cache), and RRF over our two lists is the same fusion family with zero quality loss and no duplicate index. (So LanceDB earns its place on columnar vectors / ANN / metadata filtering — not native hybrid.)
Reranking is a first-class stage, wired in v1 behind a toggle. A local cross-encoder (e.g. bge-reranker-v2-m3 via the ONNX runtime already shipped for embeddings — no new stack, no content leaving the box per externals.md) re-scores the fused top-N. It's the biggest quality lever and matters most in lode's regime (small corpus, short queries), and cited Q&A lives or dies on ranking. The seam is non-negotiable (painful to retrofit); the model is swappable/disableable so it can be A/B'd once there's a real corpus. Don't tune rerank models/thresholds pre-data (decisions.md).

The final trust_rank step applies the trust gradient — your note > your annotation > current external snapshot > stale external snapshot > AI-inferred edge — defined in externals.md. The user's own words are highest-trust; externals corroborate, they do not override.

Pinned-note context is deliberately unbounded (lode-ol2v)¶

"Ask about THIS note" (lode.retrieval.pinned_note_context, lode-35nu.11.3) pins every live-head passage of the note ahead of normal corpus retrieval, bypassing retrieval_top_k / rerank_keep_n — on purpose. The entry point's own design intent is that the pinned note is "primary context rather than competing for retrieval rank"; capping it would silently drop passages from the very note the user chose to ask about, which defeats the guarantee the feature exists to make. So: no cap, and no knob — the prompt cost grows with note length, which is the expected shape for "ask about this note" (a note long enough for this to matter is long enough that the user is deliberately trading prompt size for completeness).

What is fixed is the redundant half of that cost. Small-to-big sends each hit's parent_block, so passages chunked from the same block used to send that block's text once per passage sharing it — pure waste, no completeness benefit. The egress boundary (cited_answer.ask) is where that is deduped, not the retrieval layer: retrieval must keep every ContextItem distinct, since each carries its own char_range and that is what pins a citation's offset. The dedup's safety argument lives next to the code in cited_answer.ask.

Faithfulness: verify citations, don't just require them¶

The primary bet is cited Q&A, and the stated value is "hallucinated synthesis is worse than none" (design.md §2). Requiring a citation field in the response schema does not deliver that — an LLM will emit a well-formed citation that doesn't support its claim. A confident answer with a plausible-but-wrong citation is the worst outcome: a hallucination wearing the uniform of a verified fact, which the user trusts more because it's cited. So citation faithfulness is enforced as a pipeline stage, not assumed from the schema.

Failure modes¶

Citation–claim mismatch — cites a note that says the opposite of the claim.
Quote fabrication — emits a "quoted" span that appears in no version.
Paraphrase drift — the note is topically right but a specific payload (number, date, decision) is wrong.
Unsupported synthesis — fuses notes A + B into a claim neither supports.

Make the answer schema verifiable¶

The Q&A LLM does not return prose + [note_id]. It returns a list of claims, each carrying the exact evidence it rests on — pinned to the version, not the logical note (bytes drift across versions), and to a span, not the whole note:

answer = [
  { text: "<one factual claim>",
    support: [ { version_id | snapshot_id,
                 quoted_span: "<verbatim text from that version>" } ] },
  ...
]

WONT-DO (lode-35nu.2): no per-claim LLM self-reported annotation or confidence. A richer answer schema — the Q&A LLM itself grouping claims and/or attaching a confidence/annotation to each one — was proposed and rejected. Reason: self-reported confidence contradicts verify-don't-trust, the same principle the faithfulness gate below exists to enforce — a model asserting its own reliability is exactly the kind of claim the gate refuses to take on trust, so asking the model to grade itself reintroduces inside the schema the trust the gate removes downstream. Claim grouping is not requested from the model either; it is derived app-side from resolved note identity (lode-35nu.1) instead. The schema above — claim text plus pinned, verbatim evidence — is deliberately the full extent of what the model is asked to assert.

The faithfulness gate (a stage, like rerank)¶

Runs app-side, after the Q&A LLM returns and before display:

Verbatim-span check (deterministic, v1). Every quoted_span must occur (exact, or normalized-whitespace) in the body of its cited version_id/snapshot_id. No model, no latency.
Extractive coupling (deterministic, v1). A fast path: if the claim's load-bearing payload lies inside the quoted span, the claim is verified outright. This is the cheap common case and stops a model pairing a real but inverted quote with a contradicting claim, or a drifted number being both quoted-verbatim and wrong. v1 operationalization: the load-bearing payload is the claim's word-token set (case-folded, letters/digits) minus a small fixed set of grammatical glue (articles + the copula "to be") — never numbers, entities, or polarity/quantity words like "off"/"no"/"all", which are exactly what drift turns on. A claim couples iff some single cited span's tokens contain its whole payload; a payload split across spans is synthesis and falls to step 3. The stopword set is deliberately minimal — omitting a word only makes coupling stricter (a faithful claim falls through to NLI rather than fast-pathing), the fail-closed direction. A claim that passes step 1 but not coupling is handed to the step-3 entailment check below, not dropped on the spot.
Entailment check (local NLI, v1 — coarse, tuning pending; lode-1k3.4). Claims that pass the span check but not extractive coupling — genuine multi-note synthesis, and legitimate paraphrase that sits outside any single span — fall through to a local NLI / cross-encoder that scores whether the cited spans jointly entail the claim. The cited spans are joined into one premise and scored against the claim text; fastembed ships no dedicated NLI model, so the reranker cross-encoder (BAAI/bge-reranker-base via TextCrossEncoder) is repurposed as the entailment scorer (its raw logit sigmoid'd to a [0, 1] probability) — verified in lode-txh.6. Above a deliberately conservative threshold (entailment_threshold, untuned) the synthesized claim is accepted; at/below it, dropped (fail-closed). This is what gives v1 real synthesis capability — "connect these two notes" gets answered, not refused. The scorer is an injectable seam (a Protocol + a lazily-loaded FastEmbed default), so it loads no model unless a claim actually reaches step 3 and tests stay offline. It runs on the same ONNX runtime as the reranker — on-box, no $, no cloud-LLM round-trip.
Drop or flag claims that fail; never silently display them.
Abstain. If nothing survives the gate, the system says "your notes don't answer this" — the honest failure mode. Fidelity over fluency means a willingness to return nothing rather than a confident hallucination.

SPIKE (lode-35nu.8): no change to _WORD or normalize_whitespace — but not because tokenization is verdict-neutral. Step 2's _WORD ([^\W_]+, maximal runs of letters/digits) is applied to both claim.text and each quoted_span, splitting contractions (doesn't → doesn+t), possessives (client's → client+s), hyphenated compounds (allkeys-lru → allkeys+lru), and parenthesized identifiers (age(datfrozenxid) → age+datfrozenxid).

Applying the same regex to both sides does not make the verdict independent of where it draws boundaries: coupling is asymmetric containment — payload(claim) ⊆ tokens(span) — so splitting more finely enlarges the span's token set and is monotonically more permissive. Counter-example, run against the shipped code: claim "the policy is allkeys-lru" against span "maxmemory-policy allkeys-lru" couples under _WORD (the span's maxmemory-policy contributes a bare policy token) but not under a hyphen/apostrophe-preserving alternative ([^\W_]+(?:[-'][^\W_]+)*). The same mechanism lets a claim naming only a fragment of a compound identifier (DNS where the span says DNS-01) couple.

That inverts the usual bounding argument. A coupling miss is harmless — it falls through to NLI entailment (step 3), which is fail-closed (gate._claim_survives returns claim_entailed(...), not True). But the tokenization risk is a spurious hit, which returns True and bypasses step 3 entirely — the fail-open direction. "A miss only costs latency" does not cover the real exposure.

Evidence and its limits. 8 claim/span pairs drawn from the eval corpus (src/lode/eval/corpus/*.md, src/lode/eval/golden.py) covering every construct above gave identical verdicts under both regexes. That is a corpus-specific null result from a throwaway harness (no standing test was added), not a proof — the counter-example shows verdict-flipping inputs exist; none appear in the corpus.

Recommendation: no change. The richer regex is a pure tightening (it only demotes claims to NLI) with zero measured benefit on the corpus; a tokenizer library (spaCy, regex with Unicode word-break rules) buys the same absent benefit at the cost of a dependency. Separately, a mismatched Unicode normalization form between an LLM-generated claim and a stored span is a narrower failure that word-splitting refinement cannot close at all.

DECISION (lode-1qxy): exposure explicitly accepted, _WORD/normalize_whitespace unchanged. Closing the "revisit" left open above: the compound-identifier fragment exposure (allkeys-lru / DNS-01) is real but stays accepted, for the same zero-measured-benefit-on-corpus reasoning as the SPIKE. A standing regression test (tests/test_faithfulness.py::test_compound_identifier_fragment_couples_known_fail_open_exposure) pins the current behavior (both examples above couple) as a canary: if _WORD is ever tightened — here or as a side effect of unrelated tokenizer work — that test fails and must be updated deliberately alongside this note, rather than the exposure silently closing or reopening unnoticed. This concerns only the fragment-containment mechanism in _WORD; other fail-open paths through claim_extractively_coupled (e.g. inverted polarity) are tracked and closed separately.

DECISION (lode-w2y7): negated-span coupling hole — closed, via a small negation-cue check. Containment (payload(claim) ⊆ tokens(span)) is asymmetric in a second, distinct way from lode-1qxy above: it never looks at tokens the span carries that the claim's payload doesn't need. A span containing a negation the claim lacks — claim "the cache is invalidated" vs span "the cache isn't invalidated" — still couples: {cache, invalidated} is a subset of the span's tokens regardless of the isn/t fragments sitting alongside them. This is fail-open: coupling returning True skips NLI entailment entirely (gate.py's _claim_survives), so the contradicting claim is verified outright instead of falling through to the fail-closed entailment check.

Fix: a negation-cue asymmetry check, additive to containment, not a change to tokenization. A span couples only if it carries no negation cue (_negation_cues in faithfulness.py) that the claim itself lacks. Two cue kinds:

Standalone negation words, matched as whole tokens from the existing _WORD tokenizer: not, no, never, cannot, nothing, nobody, none, neither, nor, without.

n't contractions, matched on the raw text (\w+n['’]t\b), not the token set. _WORD splits on the apostrophe and leaves only a stem behind (isn't → isn+t), so a stem list would have to either collide with real words (don a name, won the past tense of "win") or miss don't/won't — two of the most common negations in English. Matching the contraction whole avoids both horns, and makes this check independent of where _WORD draws its boundaries, so it neither depends on nor constrains lode-1qxy. Both the ASCII and the typographic apostrophe are accepted; stored bodies are web prose as often as typed notes.

A cue present in the claim itself (a claim that states its own negation) does not block coupling — only a cue asymmetric to the span does.

Residual exposure, stated plainly. This is a lexical cue check, not a polarity parser. It does not see affixal negation (unchanged, non-empty), multiword hedges (fails to, hardly, no longer beyond the bare no), or negation scope — a span may negate a clause the claim never touched. Every such miss leaves the original fail-open behaviour intact for that input: the claim fast-path-couples and skips NLI, exactly as before the fix. The fix strictly shrinks that exposure and never widens it, because cues are only ever added to the blocking side — but it does not eliminate it, and the remainder is bounded only by how much negation English expresses lexically. Revisit if the eval harness exhibits it; the same "record rather than dismiss" register as the lode-1qxy note above.

Why this option, and why it needed no escalation. The ticket offered three options; this is option 1 (a cue list), taken because it is the only one with no tradeoff to weigh: it only removes fast-path couplings (the fail-closed direction — a demoted claim goes to NLI, never straight to verified), so it cannot widen the gate. Option 2 (bounding unmatched span tokens) would have demoted many currently-fast-path claims and carries a real latency cost — that one is a judgment call for a human. Standing regression tests live in tests/test_faithfulness.py (test_negated_span_is_not_coupled and the don't/won't, typographic-apostrophe, and don/won-as-real-words cases), and nothing here touches _WORD, normalize_whitespace, or any tokenization boundary — lode-1qxy remains open and separately tracked.

The gate verifies each span against the stored bytes of the cited version/snapshot, resolved only for the egress-cleared targets — the same set eligible to reach the model. A no-egress target's body is withheld from that verification set, so a claim citing content the model was never shown fails closed and is dropped, exactly like a fabricated quote; the withheld item still surfaces as "present, withheld from cloud synthesis" rather than being silently lost.

The entailment threshold ships untuned and must be revisited. Steps 1–2 are deterministic and need no tuning. Step 3's model choice and acceptance threshold are a real knob that cannot be set honestly without a corpus: shipped too loose, it readmits unsupported synthesis (mode 4); too tight, it collapses back to extractive-only. v1 ships it conservative and fail-closed so the capability exists from day one, and the model + threshold are tuned against the eval harness once there's real Q&A data (decisions.md). Treat v1 synthesis answers as capability-present, quality-untuned.

What v1 catches — and what it doesn't¶

The gate fails closed: any claim it cannot verify (deterministically or by entailment) is dropped, and if nothing survives, the system abstains. So the safety question ("could a hallucination ship?") and the capability question ("can a valid claim of this kind get answered?") are separate — and the table says which it means. v1 is safe on all four modes, and has the capability for all four — with synthesis quality gated on tuning.

Failure mode	Shipped to user in v1? (safety)	Answerable in v1? (capability)
2 · Quote fabrication	Never — quote isn't in the version, dropped	n/a
1 · Citation–claim mismatch	Almost never — extractive coupling rejects a quote that doesn't contain the claim	yes
3 · Paraphrase drift	Almost never — caught by extractive coupling, or by the entailment check when the paraphrase sits outside a span	yes
4 · Unsupported synthesis	Conservatively — the entailment gate admits a composed claim only when the cited spans jointly entail it above threshold; below, dropped	Yes (coarse) — synthesis is answered via the entailment gate; quality depends on a threshold that ships untuned and is revisited post-corpus

Mode 4 is now handled by validation, not blanket abstention: the entailment check lets valid synthesis through while still catching invalid synthesis. The residual risk is concentrated in one place — the untuned threshold — which is why it's called out as a revisit, not a settled value. An optional LLM-judge second pass can serve as a "high-assurance" toggle (stronger than local NLI), but it costs a round-trip + dollars per answer and re-ships content off-box, so it is not the default.

Tool-augmented Ask: the tool path is the draw-down path¶

Ask can reach live external systems mid-answer — read-only JIRA/Confluence/web tools the LLM may call when the corpus doesn't already hold what the question needs (lode-35nu.11). The governing constraint is that this must not become a second, ungated evidence path: a live tool result has no version_id/snapshot_id, so the faithfulness gate above has no stored bytes to verify a span against. The answer is not to weaken the gate or to add an "unverified claims" section — it is to make tool results into ordinary evidence before synthesis sees them.

Snapshot-then-cite. Every tool result that will be cited is persisted on the existing externals/snapshots path (content-addressed body, fetched_at, provenance, no_egress evaluated) before it is handed to synthesis, reusing lode.drawdown / lode.externals rather than inventing a second snapshot writer. Synthesis then cites it by snapshot_id like any other external, cited_answer._resolve_targets verifies spans against stored bytes, and all five gate steps run unmodified. The cost is a write per fetched resource, and it is accepted.

Three consequences of that constraint are settled in externals.md (lode-35nu.11.5) rather than restated here, because they are facts about external identity, persistence, and egress, not about retrieval:

Search returns identifiers, never bodies. A query result has no stable external_id, so it is never persisted and never citable; the model fetches the specific resources it names, and those are snapshotted and cited. Discovery is not citation.
Ask-time snapshots are first-class externals rows marked discovered_via = 'ask', with no note→external edge. No ephemeral class, no separate GC; dedup and identical-refetch suppression fall out of the existing external_id / snapshot_id machinery.
A tool call is an egress event — it writes an egress_log row (purpose = 'tool'), gate_qa_egress's redaction runs over the tool arguments and not only over passages, and a tool call whose destination source is no_egress is forbidden.

The gate is the reason the whole sub-tree is shaped this way: everything above exists so that a claim sourced from a live lookup is exactly as verifiable as a claim sourced from a note the user wrote a year ago, and reaches the user through the identical stage.