Benchmarking Retrieval Representations for Question Answering over Construction Drawing Sets
65.8% Precision@1
Research summary
65.8% Precision@1 by fusing text, sheet summaries, and visual retrieval.
Research record
This record preserves the title and author order of the supplied manuscript. Its research stage remains separate from the existence of a PDF or conference-formatted layout.
Evaluation results, cohort definitions, limitations, and source locators are maintained in the related project. This record does not duplicate the project metrics.
This conference-form record shares the drawing-retrieval benchmark with the journal-form record. The relationship does not assert which manuscript came first or that either venue accepted it.
This record shares a benchmark with the related manuscript version. It is not an independent replication.
Selected finding
The headline result from the related case study, with its sample and source.
Full retrieval benchmark
407 questions. 434 sheets, three projects, seventeen configurations; 86 scanned sheets.
Precision@1
Reported in sourceHighest reported value: Triple fusion, 65.8%.
Data table
| Condition | Value | Uncertainty |
|---|---|---|
| Raw sheet BM25 | 45.0% | 95% CI [40.0, 49.6] |
| Card BM25 | 56.5% | 95% CI [51.6, 61.2] |
| ColQwen2 | 59.0% | 95% CI [54.3, 63.6] |
| Triple fusion | 65.8% | 95% CI [61.2, 70.3] |
Denominator: 407 human-authored questions; first retrieved sheet relevance.
Adapted from Local Multimodal Retrieval-Augmented Question Answering over Construction Drawing Sets, Table 3; Section 4.2; Figure 11.
Evaluation Scope & Methodology
Samples, denominators, uncertainty, sources and limits
Approach
Use a shared sheet-localization harness with source-reported bootstrap confidence intervals; distinguish full retrieval from exploratory selective retrieval.
Results and scope belong to the related case study, Local multimodal drawing retrieval.
Cohorts and samples
| Cohort | Sample | Definition |
|---|---|---|
| Full retrieval benchmark | 407 questions | 434 sheets, three projects, seventeen configurations; 86 scanned sheets. |
| Exploratory answered subset | 241 questions | Threshold selected on the full evaluation questions; 241/407 answered. |
Reported measures
| Measure | Unit | Cohort | Denominator | Uncertainty | Evidence |
|---|---|---|---|---|---|
| Precision@1 | Percent | Full retrieval benchmark | 407 human-authored questions; first retrieved sheet relevance | Raw sheet BM25: 95% CI [40.0, 49.6]; Card BM25: 95% CI [51.6, 61.2]; ColQwen2: 95% CI [54.3, 63.6]; Triple fusion: 95% CI [61.2, 70.3] | Reported in source |
| Full-set Recall@10 | Percent | Full retrieval benchmark | 407 questions; every required sheet must occur in the first ten results | None reported | Reported in source |
| Exploratory selective Precision@1 | Percent | Exploratory answered subset | 241 answered questions from the same 407 used to choose the threshold | None reported | Reported in source |
| Mean reciprocal rank | Score from 0 to 1 | Full retrieval benchmark | All 407 questions, triple-fusion pipeline | Triple fusion: 95% CI [0.736, 0.802] | Reported in source |
| Selective retrieval coverage | Percent | Exploratory answered subset | 241 accepted queries out of 407 evaluation queries | None reported | Reported in source |
Uncertainty definitions
- 95% CI. 95% confidence interval as reported by the source.
Limitations
- Sheet localization is distinct from end-to-end answer accuracy.
- The selective threshold was selected on the evaluation questions.
- Reported runtime uses a 48 GB Apple-Silicon desktop; a 12 GB GPU configuration has not been established.
- The conference and journal forms share a benchmark and are not independent replications.
Sources
- Local Multimodal Retrieval-Augmented Question Answering over Construction Drawing Sets, Table 3; Section 4.2; Figure 11 (Supplied manuscript reviewed November 2026). Public release pending
- Benchmarking Retrieval Representations for Question Answering over Construction Drawing Sets, Abstract; benchmark description (Supplied manuscript reviewed November 2026). Public release pending
Cite this record
Muhammad Tahir Khan, Asif Faisal Chowdhury, Obiora Odugu, Chao Wang. Benchmarking Retrieval Representations for Question Answering over Construction Drawing Sets. Manuscript.
Formatted from the supplied manuscript. Publisher metadata has not been verified, so no BibTeX is offered yet.