Local multimodal drawing retrieval
65.8% Precision@1
65.8% Precision@1 by fusing text, sheet summaries, and visual retrieval.
My contribution
Co-authored evaluation of local retrieval representations and fusion for construction drawing sets.
Find the sheet before answering
A drawing question can depend on a detail, schedule, or cross-reference buried in hundreds of sheets. This work evaluates sheet localization before answer generation. The benchmark contains 434 sheets and 407 human-authored questions from three projects, including 86 scanned sheets.
The index separates native text or OCR, structured summary cards, and visual multivectors. Query-time retrieval compares seventeen configurations, including sparse text search, visual retrieval, fusion, graph expansion, and reranking. The best triple fusion combines card BM25, raw-text BM25, and ColQwen2 visual retrieval.
What the metrics mean
Precision@1 asks whether the first returned sheet is relevant. Full-set Recall@10 asks whether all required sheets occur in the first ten results. Triple fusion reports 65.8% Precision@1 and 81.8% full-set Recall@10. These numbers describe different criteria and should not be averaged into a generic accuracy score.
An exploratory selective operating point answers 241 of 407 questions at 80.1% Precision@1. Its threshold was tuned on these same questions. A deployment claim needs a held-out threshold evaluation, and an empirical risk–coverage curve needs query-level exports.
Two manuscript forms, one benchmark
The journal and conference records remain separately citable, with their own titles and author lists. Their relationship denotes a shared experiment. It does not establish submission order, acceptance, or a chronological derivative relationship.
System architecture
Select a step, or a block in the drawing, to see its input, computation, output and source.
Index once, retrieve three ways, fuse
Index time builds raw-text, summary-card and visual indexes. Query time searches all three and fuses the rankings. The benchmark scores the ranked sheets, not generated answers.
- Lexical (BM25) and fusion
- Visual model
- Evaluated alternative
- Outside scored scope
Drawing sheetsInput
- Input
- Construction drawing sets from three projects, including scanned sheets.
- Computation
- None.
- Output
- Sheet pages for indexing.
- Source
- Drawing retrieval journal manuscript, Figures 1 and 2; conference manuscript, Figure 1
Native text or OCRProcessing step
- Input
- Sheet pages.
- Computation
- Uses native PDF text, and OCR for scanned sheets.
- Output
- Raw sheet text.
- Source
- Drawing retrieval journal manuscript, Figures 1 and 2; conference manuscript, Figure 1
Structured summary cardsProcessing step
- Input
- Sheet content.
- Computation
- Builds a structured summary card for each sheet.
- Output
- Card text.
- Source
- Drawing retrieval journal manuscript, Figures 1 and 2; conference manuscript, Figure 1
Page imagesProcessing step
- Input
- Sheet pages.
- Computation
- Renders each sheet as an image.
- Output
- Images for visual indexing.
- Source
- Drawing retrieval journal manuscript, Figures 1 and 2; conference manuscript, Figure 1
Raw-text BM25 indexDeterministic
- Input
- Raw sheet text.
- Computation
- Sparse lexical index.
- Output
- Searchable raw-text index.
- Source
- Drawing retrieval journal manuscript, Figures 1 and 2; conference manuscript, Figure 1
Card BM25 indexDeterministic
- Input
- Card text.
- Computation
- Sparse lexical index over summary cards.
- Output
- Searchable card index.
- Source
- Drawing retrieval journal manuscript, Figures 1 and 2; conference manuscript, Figure 1
ColQwen2 multivectorsModel
- Input
- Page images.
- Computation
- Visual multivector embeddings.
- Output
- Visual index.
- Source
- Drawing retrieval journal manuscript, Figures 1 and 2; conference manuscript, Figure 1
QuestionInput
- Input
- A human-authored question about the drawing set.
- Computation
- None.
- Output
- Query text.
- Source
- Drawing retrieval journal manuscript, Figures 1 and 2; conference manuscript, Figure 1
Card BM25 retrievalDeterministic
- Input
- Query text.
- Computation
- Lexical search over the card index.
- Output
- Ranked sheets.
- Source
- Drawing retrieval journal manuscript, Figures 1 and 2; conference manuscript, Figure 1
Raw-text BM25 retrievalDeterministic
- Input
- Query text.
- Computation
- Lexical search over the raw-text index.
- Output
- Ranked sheets.
- Source
- Drawing retrieval journal manuscript, Figures 1 and 2; conference manuscript, Figure 1
ColQwen2 visual retrievalModel
- Input
- Query text.
- Computation
- Late-interaction search over visual multivectors.
- Output
- Ranked sheets.
- Source
- Drawing retrieval journal manuscript, Figures 1 and 2; conference manuscript, Figure 1
Rank fusionDeterministic
- Input
- Three ranked lists.
- Computation
- Fuses the three rankings into one (triple fusion).
- Output
- One ranked sheet list.
- Source
- Drawing retrieval journal manuscript, Figures 1 and 2; conference manuscript, Figure 1
Ranked sheetsOutput
- Input
- Fused ranking.
- Computation
- Scored by Precision@1, MRR and full-set Recall@10.
- Output
- Sheet localization result.
- Source
- Drawing retrieval journal manuscript, Figures 1 and 2; conference manuscript, Figure 1
Graph expansion and rerankingComparison path
- Input
- Retrieved sheets.
- Computation
- Evaluated alternative configurations among the seventeen.
- Output
- Comparison results, not an assumed improvement.
- Source
- Drawing retrieval journal manuscript, Figures 1 and 2; conference manuscript, Figure 1
Answer generationOutside evaluated scope
- Input
- Retrieved sheets and the question.
- Computation
- Generates an answer.
- Output
- Not scored by the sheet-localization metrics.
- Source
- Drawing retrieval journal manuscript, Figures 1 and 2; conference manuscript, Figure 1
Adapted from Local Multimodal Retrieval-Augmented Question Answering over Construction Drawing Sets, Table 3; Section 4.2; Figure 11; and Benchmarking Retrieval Representations for Question Answering over Construction Drawing Sets, Abstract; benchmark description.
Both manuscript forms describe one shared benchmark. The website serves precomputed results and runs no model inference.
Animated pulses show conceptual signal flow only. They do not show measured execution time or a live inference run.
Findings
Reported results, grouped by sample. Each figure keeps its own scale and links to its source and method.
Full retrieval benchmark
407 questions. 434 sheets, three projects, seventeen configurations; 86 scanned sheets.
Precision@1
Reported in sourceHighest reported value: Triple fusion, 65.8%.
Data table
| Condition | Value | Uncertainty |
|---|---|---|
| Raw sheet BM25 | 45.0% | 95% CI [40.0, 49.6] |
| Card BM25 | 56.5% | 95% CI [51.6, 61.2] |
| ColQwen2 | 59.0% | 95% CI [54.3, 63.6] |
| Triple fusion | 65.8% | 95% CI [61.2, 70.3] |
Denominator: 407 human-authored questions; first retrieved sheet relevance.
Adapted from Local Multimodal Retrieval-Augmented Question Answering over Construction Drawing Sets, Table 3; Section 4.2; Figure 11.
Full-set Recall@10
Reported in sourceFull-set Recall@10, Triple fusion: 81.8%.
Data table
| Condition | Value |
|---|---|
| Triple fusion | 81.8% |
Denominator: 407 questions; every required sheet must occur in the first ten results.
Adapted from Local Multimodal Retrieval-Augmented Question Answering over Construction Drawing Sets, Table 3; Section 4.2; Figure 11.
Mean reciprocal rank
Reported in sourceMean reciprocal rank, Triple fusion: 0.769 (95% CI [0.736, 0.802]).
0.769Triple fusion95% CI [0.736, 0.802]
Denominator: All 407 questions, triple-fusion pipeline.
Adapted from Local Multimodal Retrieval-Augmented Question Answering over Construction Drawing Sets, Table 3; Section 4.2; Figure 11.
Exploratory answered subset
241 questions. Threshold selected on the full evaluation questions; 241/407 answered.
Exploratory selective Precision@1
Reported in sourceExploratory selective Precision@1, Selective retrieval: 80.1%.
Data table
| Condition | Value |
|---|---|
| Selective retrieval | 80.1% |
Denominator: 241 answered questions from the same 407 used to choose the threshold.
Adapted from Local Multimodal Retrieval-Augmented Question Answering over Construction Drawing Sets, Table 3; Section 4.2; Figure 11.
Selective retrieval coverage
Reported in sourceSelective retrieval coverage, Selective setting: 59.2%.
Data table
| Condition | Value |
|---|---|
| Selective setting | 59.2% |
Denominator: 241 accepted queries out of 407 evaluation queries.
Adapted from Local Multimodal Retrieval-Augmented Question Answering over Construction Drawing Sets, Table 3; Section 4.2; Figure 11.
Evaluation Scope & Methodology
Samples, denominators, uncertainty, sources and limits
Approach
Use a shared sheet-localization harness with source-reported bootstrap confidence intervals; distinguish full retrieval from exploratory selective retrieval.
Cohorts and samples
| Cohort | Sample | Definition |
|---|---|---|
| Full retrieval benchmark | 407 questions | 434 sheets, three projects, seventeen configurations; 86 scanned sheets. |
| Exploratory answered subset | 241 questions | Threshold selected on the full evaluation questions; 241/407 answered. |
Reported measures
| Measure | Unit | Cohort | Denominator | Uncertainty | Evidence |
|---|---|---|---|---|---|
| Precision@1 | Percent | Full retrieval benchmark | 407 human-authored questions; first retrieved sheet relevance | Raw sheet BM25: 95% CI [40.0, 49.6]; Card BM25: 95% CI [51.6, 61.2]; ColQwen2: 95% CI [54.3, 63.6]; Triple fusion: 95% CI [61.2, 70.3] | Reported in source |
| Full-set Recall@10 | Percent | Full retrieval benchmark | 407 questions; every required sheet must occur in the first ten results | None reported | Reported in source |
| Exploratory selective Precision@1 | Percent | Exploratory answered subset | 241 answered questions from the same 407 used to choose the threshold | None reported | Reported in source |
| Mean reciprocal rank | Score from 0 to 1 | Full retrieval benchmark | All 407 questions, triple-fusion pipeline | Triple fusion: 95% CI [0.736, 0.802] | Reported in source |
| Selective retrieval coverage | Percent | Exploratory answered subset | 241 accepted queries out of 407 evaluation queries | None reported | Reported in source |
Uncertainty definitions
- 95% CI. 95% confidence interval as reported by the source.
Limitations
- Sheet localization is distinct from end-to-end answer accuracy.
- The selective threshold was selected on the evaluation questions.
- Reported runtime uses a 48 GB Apple-Silicon desktop; a 12 GB GPU configuration has not been established.
- The conference and journal forms share a benchmark and are not independent replications.
Sources
- Local Multimodal Retrieval-Augmented Question Answering over Construction Drawing Sets, Table 3; Section 4.2; Figure 11 (Supplied manuscript reviewed November 2026). Public release pending
- Benchmarking Retrieval Representations for Question Answering over Construction Drawing Sets, Abstract; benchmark description (Supplied manuscript reviewed November 2026). Public release pending