← Articles & research
Technical report · Long-document retrieval

Source-grounded retrieval for long documents.

A one-line-per-article map turns LLM query rewriting into routing: 85.2% passage recall on paraphrased questions, reading 2,000 tokens of a 3-million-token book per query.

Abstract

Sending a whole book with every question is costly and capped by the context window. Thumbindex (working name) keeps documents as written, indexes them with cheap content-derived addresses, and re-reads the source before returning evidence. On a 3,256-page encyclopedia (≈3M tokens) it places the gold passage inside a 2,000-token budget for 77.9% of 308 test questions (BM25: 48.1%). The main result: giving the query-rewriting model a map of the corpus, one generated line per article, adds 14.8 points on paraphrased questions with no vectors (95% CI 10.0 to 19.3) and 5.2 points on top of hybrid retrieval with reranking (CI 1.7 to 8.6), reaching 85.2%. Content-hashed spans keep every pointer valid after an edit.

85.2%
Paraphrased-question recall: map + hybrid retrieval.
+14.8
Points from the map alone, with no vector index.
0.07%
Of the book read per query: 2,000 of ~3M tokens.
100%
Pointers valid after a paragraph insert (207 of 207).

1. System

Documents are the source of truth. Retrieval runs over small derived structures; every candidate is re-read from the store and hash-checked before it is returned, so a stale index entry can cost recall but never return text that is not in the source.

  • Blocks. Paragraphs named by content hash. A version is the ordered list of its block hashes.
  • Spans. ~200-word runs of blocks with content-defined boundaries; an edit moves boundaries only near itself.
  • Addresses. Lexical fields, heading path, position, source and named entities per span, scored by BM25 and floor-bounded soft cue scorers.
  • Map. One model-written line per article, built at ingest and placed in the rewrite prompt, where it is cacheable.
  • Levels. L0: BM25 + soft cues, standard library only. L1: + an LLM rewrite fused with the original query. L2: + a local dense embedder and cross-encoder reranker.
System architectureSources are ingested into paragraph blocks named by content hash, roughly 200-word spans with addresses, and a one-line-per-article map. A question is rewritten with the map in view, searched over addresses, re-read from the stored source with a hash check, and answered with its address.1 · BUILD Once per source, and again for every edit Sources Books PDF, text Notes Markdown Team wikis Confluence NEXT Ingest 1 Cut the text into paragraphs, each named by its content 2 Group them into passages of about 200 words 3 Give every passage its addresses 4 Write one line per article for the map TESTED When a page changes Compare by content Redo only changes Keep the old versions Pointers stay valid 2 · STORE Kept on your machine Map One line per article, written by a model TESTED Addresses Words, headings, place, author + VECTORS AT LEVEL 2 Source Kept as written, every version Traces Questions asked, on this device 3 · ASK Every question Question Your words, plus what you remember Rewrite The model sees the map LEVEL 1 AND UP Search Ranks passages by words and cues + VECTORS AT LEVEL 2 Re-read About 2,000 tokens of source, checked by hash Answer Any model, with page and paragraph
Figure 1. Architecture. Each stored structure (dark) feeds the query step directly below it. TESTED: measured in experiments, not yet in the library. NEXT: planned. Full diagram (PNG).
How a question is answered
How a question is answered A question is rewritten by a model that sees the map, the passages are ranked using the addresses, about 2,000 tokens of the source are re-read, and the answer is given with its page and paragraph. THE QUESTION WHAT IS STORED Question Rewrite Search Re-read Answer In your own words The model sees the map Rank the passages 2,000 tokens of source With page and paragraph Map Addresses Source One line per article Words, section, place Kept as written
Figure 1. The query path and the stored structure each step reads. Full diagram (PNG).

2. Setup

3,256
PDF pages, two columns per page.
≈3M
Tokens (2.0M words) in one technical encyclopedia.
555
Articles, one document each.
13,756
Spans of about 200 words.
308 + 420
Test questions: whole book, and paraphrased over a 185-article core.
2,000
Tokens read per question, at most.
Primary metric

Evidence hit rate @2k: did the system actually read the passage that supports the answer, within 2,000 tokens? Answer fluency is not scored.

QuestionsWhole book: 57 lexical-overlap, 92 paraphrased, 159 vague. Core: 420 paraphrased over 185 articles (≈1M tokens).
StatisticsPre-registered contrasts and minimum effects, paired bootstrap 95% CIs, Holm correction. Test questions never reach an indexing step.
ModelsDeepSeek chat API for rewrites and map lines. Local embedder and cross-encoder reranker at level 2.

3. Results

3.1 Paraphrased questions are where retrieval breaks

Every method solves lexical-overlap questions. Paraphrases are the gap: BM25 finds 13.0% of them, and hybrid retrieval with a cross-encoder reranker still misses 39%.

Evidence hit rate @2k by question type
308 test questions over the whole book (≈3M tokens).
All questions308 questions
BM25
48.1%
BM25 + LLM rewrite
67.9%
Hybrid + reranker
76.6%
Hybrid + LLM rewrite
75.0%
Hybrid + cues (RRF)
80.8%
Word overlap57 questions
BM25
98.2%
BM25 + LLM rewrite
94.7%
Hybrid + reranker
96.5%
Hybrid + LLM rewrite
86.0%
Hybrid + cues (RRF)
98.2%
Paraphrased92 questions
BM25
13.0%
BM25 + LLM rewrite
32.6%
Hybrid + reranker
60.9%
Hybrid + LLM rewrite
52.2%
Hybrid + cues (RRF)
62.0%
Vague159 questions
BM25
50.3%
BM25 + LLM rewrite
78.6%
Hybrid + reranker
78.6%
Hybrid + LLM rewrite
84.3%
Hybrid + cues (RRF)
85.5%
Figure 2. Hybrid: BM25 and dense retrieval fused, cross-encoder reranking, heading-path prefix. Rewrite: the LLM's rewrite replaces the query. Cues (RRF): cue rankings merged by reciprocal rank fusion.
MethodAllWord overlapParaphrasedVague
BM2548.198.213.050.3
BM25 + LLM rewrite67.994.732.678.6
Hybrid + reranker76.696.560.978.6
Hybrid + LLM rewrite75.086.052.284.3
Hybrid + cues (RRF)80.898.262.085.5
Table 1. Evidence hit rate @2k (%), whole book. Best per column in bold.
Why the remaining misses happened
50 failures from the dev split, read and sorted by cause.
Paraphrased, no cue
27
Paraphrased, no cue, loosely worded
10
Built-in wrong cue
5
Only position or topic cues
4
Cue corrupted by noise
3
Word-overlap question
1
Figure 3. 37 of 50 failures (74%) were paraphrased questions with no cue to go on. That is the case the map targets.

3.2 A one-line map turns rewriting into routing

A plain LLM rewrite lifts paraphrase recall only from 41.9% to 51.0%, because the model knows neither the corpus vocabulary nor where a topic lives. With the map in its prompt, the rewrite names the right article and uses its terms.

Evidence hit rate @2k, paraphrased questions
420 test questions, 185-article core (≈1M tokens).
BM25
41.9%
BM25 + LLM rewrite
51.0%
BM25 + hypothetical passage
59.5%
BM25 + map-conditioned rewrite
65.7%
Dense retrieval alone
66.0%
Hybrid + reranker + rewrite
80.0%
Hybrid + reranker + map-conditioned rewrite
85.2%
Figure 4. The map adds 14.8 points to the plain rewrite (95% CI 10.0 to 19.3) and 5.2 points to the full stack (CI 1.7 to 8.6, Holm-corrected p = 0.015).
  • Vector-level recall without vectors. BM25 with a map-conditioned rewrite (65.7%) matches dense retrieval alone (66.0%).
  • Additive with vectors. On top of hybrid retrieval, reranking and a rewrite, the map adds 5.2 points, to 85.2%.
  • The mechanism is routing. All of the gain comes from the 250 of 420 questions where the rewrite names the right article.
  • Ahead of an agent. On 150 questions, an agent that searched with BM25 and read pages (1.6 tool calls per question) reached 54.0%. Level 2 reached 82.7%, and level 2 with the map 88.0%.

3.3 The map at full scale

Over all 555 articles the flat map is about 24,500 prompt tokens per query, nearly all served from the prompt cache.

Evidence hit rate @2k, paraphrased questions, whole book
555 articles (≈3M tokens). Cost as billed with prompt caching.
BM25
32.1%
LLM rewrite, no map
43.3%
Two-step map4,530 prompt tokens · $0.00041 · 2.8 s
53.8%
Flat map24,532 prompt tokens · $0.00034 · 1.9 s
58.6%
Show the numbers as a table
MethodHit @2kPrompt tokensBilledUncachedLatency
BM2532.1%————
LLM rewrite, no map43.3%————
Two-step map53.8%4,530$0.00041$0.00142.8 s
Flat map58.6%24,532$0.00034$0.00741.9 s
Figure 5. The flat map adds 15.3 points over the plain rewrite. The two-step map misroutes 39% of questions and trails the flat map by 4.8 points (95% CI 0.7 to 9.0).

3.4 Three levels behind one interface

Measured through the library's own interface on the 308 whole-book questions, with names, authors and sections corrupted 30% of the time:

Evidence hit rate @2k by level and question type
Whole book (≈3M tokens).
All questions308 questions
BM25
48.1%
Level 0 · no model
53.6%
Level 1 · + rewrite
71.1%
Level 2 · + dense, reranker
77.9%
Word overlap57 questions
BM25
98.2%
Level 0 · no model
98.2%
Level 1 · + rewrite
98.2%
Level 2 · + dense, reranker
94.7%
Paraphrased92 questions
BM25
13.0%
Level 0 · no model
14.1%
Level 1 · + rewrite
31.5%
Level 2 · + dense, reranker
63.0%
Vague159 questions
BM25
50.3%
Level 0 · no model
60.4%
Level 1 · + rewrite
84.3%
Level 2 · + dense, reranker
80.5%
Figure 6. Level 2 lifts paraphrased questions from 13.0% to 63.0%. Level 0 needs nothing beyond the Python standard library.
LevelAddsAllLatency
0BM25 + soft cues, no model53.6%17 ms
1Fused LLM rewrite71.1%23 ms + call
2Dense model + reranker77.9%3.0 s
Table 2. Hit rate on all 308 questions and median latency per question.

3.5 Content-addressed spans survive edits

A pointer names what a passage says; its position is only a hint.

One paragraph inserted at the top of a page
A paragraph inserted at the top of a pageVersion 1 has five paragraphs. Version 2 adds a new paragraph at the top. Each old paragraph keeps its hash and is matched to the same paragraph in version 2, while every character offset moves by 312. VERSION 1 VERSION 2 ¶ 3f9a chars 0–812 ¶ c71e chars 813–1,540 ¶ 08d4 chars 1,541–2,205 ¶ b52f chars 2,206–3,010 ¶ e6a0 chars 3,011–3,640 ¶ 51c0 · new chars 0–311 ¶ 3f9a chars 312–1,124 ¶ c71e chars 1,125–1,852 ¶ 08d4 chars 1,853–2,517 ¶ b52f chars 2,518–3,322 ¶ e6a0 chars 3,323–3,952 Paragraph hash: 5 of 5 pointers still find their text Character offset: 0 of 5, every range moved by 312
A paragraph inserted at the top of a pageVersion 1 has five paragraphs. Version 2 adds a new paragraph at the top. Each old paragraph keeps its hash and is matched to the same paragraph in version 2, while every character offset moves by 312. VERSION 1 VERSION 2 ¶ 3f9a chars 0–812 ¶ c71e chars 813–1,540 ¶ 08d4 chars 1,541–2,205 ¶ b52f chars 2,206–3,010 ¶ e6a0 chars 3,011–3,640 ¶ 51c0 · new chars 0–311 ¶ 3f9a chars 312–1,124 ¶ c71e chars 1,125–1,852 ¶ 08d4 chars 1,853–2,517 ¶ b52f chars 2,518–3,322 ¶ e6a0 chars 3,323–3,952 Paragraph hash: 5 of 5 still found Character offset: 0 of 5, all moved by 312
Figure 7. Illustration with made-up hashes and offsets. Matching by paragraph hash keeps every old pointer; character ranges all shift.
Addresses still resolving to the same text after an insert
One paragraph added at the top of each page. War and Peace, 365 chapters as pages.
Character offsets
0%
Ordinal span number
53%
Span hash, greedy chunking
67%
Span hash, content-defined chunking
84%
Paragraph hash
100%
Figure 8. Ordinal pointers fail silently: a stale number resolves to different text with no error.
  • Real articles. 207 of 207 pointers resolved to the same text, with no model call.
  • Incremental re-indexing. 10 edited articles cost 80 model requests. All 4,778 entries on unchanged text were kept; stale ones were retired.

3.6 Wrong cues have a bounded cost

Cues (names, author, section, rough position) are scored softly with a floor: a wrong cue lowers a score but never excludes a passage.

Points gained by adding cues, as more of them are wrong
159 vague test questions, whole book. Bars are 95% intervals where computed.
Points gained by cues as more of them are corruptedCue arm minus baseline: +8.8 with exact cues, +8.2 at 15% corruption, -0.6 at 30%, 0.0 at 60%. With a fused rewrite on both sides: +6.3, +5.0, +0.6 and -1.3. PRE-REGISTERED −5 0 +5 +10 0% 15% 30% 60% Share of cues corrupted Points vs. baseline +8.8 +8.2 −0.6 0.0 +6.3 +5.0 +0.6 −1.3
Points gained by cues as more of them are corruptedCue arm minus baseline: +8.8 with exact cues, +8.2 at 15% corruption, -0.6 at 30%, 0.0 at 60%. With a fused rewrite on both sides: +6.3, +5.0, +0.6 and -1.3. PRE-REGISTERED −5 0 +5 +10 0% 15% 30% 60% Share of cues corrupted Points vs. baseline +8.8 +8.2 −0.6 0.0 +6.3 +5.0 +0.6 −1.3
Cues corruptedCue arm − baselineBoth with fused rewrite
0% (exact)+8.8 (4.4 to 13.8)+6.3
15%+8.2 (3.1 to 13.8)+5.0 (0.6 to 9.4)
30% (pre-registered)−0.6 (−5.7 to 4.4)+0.6 (−4.4 to 5.7)
60%0.0−1.3
Figure 9 and Table 3. Gains hold up to 15% corruption and vanish by 30%. No setting falls measurably below its baseline.
Primary result

Cues pay off only when they are accurate, and soft scoring makes wrong ones cheap. The robust gain comes from routing, not from remembered details.

3.7 What did not help

IdeaResult
Surprise-weighted landmark index50.2%, against 51.7% for random landmarks
Twelve zero-call index-key types, including the book's own back-of-book indexBest +2.6 over BM25; the pre-registered bar was +5
Replacing the query with its rewrite−1.6 on the hybrid; fusing it with the original gives +3.6
Map-conditioned hypothetical passage65.7%, no gain over the map-conditioned rewrite
Table 4. Ideas tested and closed for this corpus.

4. Scope

One corpus; questions generated by the same model family that writes the map. On the 205 of 420 paraphrased questions that pass a strict single-passage check, the vector-free map arm comes within 8.3 points of the hybrid (95% CI 2.4 to 14.1). Total model cost of all experiments: $8.15 at list prices.

Takeaway

On paraphrased questions, retrieval is largely a routing problem. A cacheable map with one line per article lets any chat model route before it searches: it matches a vector index on its own, stacks with one to 85.2%, and every answer is still read from the source.