ConstructionQA: reading supplied drawing context
A task-specific benchmark for multimodal answers about construction drawings.
My contribution
Co-authored evaluation of information extraction, sheet references, and cross-discipline coordination.
Three drawing tasks
ConstructionQA examines information extraction, sheet references, and cross-discipline coordination using 120 question–answer pairs. Its drawing set spans approximately 150+ sheets and eight disciplines. This question set is distinct from the 407-question local retrieval benchmark.
Context changes the task
Table 1 compares supplied context with distractors against guaranteed relevant context. GPT-5.4 information-extraction F1-BoW changes from 0.563 to 0.595, while Gemini 2.5 Flash changes from 0.553 to 0.610. Sheet-reference F1 improves for the listed models, but GPT-5.4’s combined cross-discipline coordination score falls from 0.563 to 0.461.
The comparison therefore needs task-specific metric names and paired context conditions. Native-PDF results belong to a separate experiment. A drawing assistant must establish both its ability to locate evidence and its ability to reason with that evidence.
Findings
Reported results, grouped by sample. Each figure keeps its own scale and links to its source and method.
ConstructionQA supplied-context questions
120 question–answer pairs. Approximately 150+ sheets across eight disciplines; task-specific subsets.
Information extraction F1-BoW, GPT-5.4
Reported in sourceHighest reported value: Guaranteed context, 0.595.
Data table
| Condition | Value |
|---|---|
| Distractors | 0.563 |
| Guaranteed context | 0.595 |
Denominator: Information-extraction task subset of the 120-pair benchmark.
Adapted from ConstructionQA: Evaluating Multimodal Large Language Models on Industrial Construction Drawing Set, Benchmark description; Table 1; Figure 2.
Information extraction F1-BoW, Gemini 2.5 Flash
Reported in sourceHighest reported value: Guaranteed context, 0.610.
Data table
| Condition | Value |
|---|---|
| Distractors | 0.553 |
| Guaranteed context | 0.610 |
Denominator: Information-extraction task subset of the 120-pair benchmark.
Adapted from ConstructionQA: Evaluating Multimodal Large Language Models on Industrial Construction Drawing Set, Benchmark description; Table 1; Figure 2.
Cross-discipline coordination combined score, GPT-5.4
Reported in sourceHighest reported value: Distractors, 0.563.
Data table
| Condition | Value |
|---|---|
| Distractors | 0.563 |
| Guaranteed context | 0.461 |
Denominator: Cross-discipline coordination task subset, Table 1.
Adapted from ConstructionQA: Evaluating Multimodal Large Language Models on Industrial Construction Drawing Set, Benchmark description; Table 1; Figure 2.
Sheet-reference F1
Reported in sourceSheet-reference F1 is higher with Guaranteed context than with Distractors for 3 of 3 entries (GPT-5.4, Gemini 2.5 Flash, and GPT-4.1 Nano).
Data table
| Entry | Condition | Value |
|---|---|---|
| GPT-5.4 | Distractors | 0.868 |
| GPT-5.4 | Guaranteed context | 0.902 |
| Gemini 2.5 Flash | Distractors | 0.801 |
| Gemini 2.5 Flash | Guaranteed context | 0.910 |
| GPT-4.1 Nano | Distractors | 0.390 |
| GPT-4.1 Nano | Guaranteed context | 0.854 |
Denominator: Same ConstructionQA sheet-reference task under distractor and guaranteed-context conditions.
Adapted from ConstructionQA: Evaluating Multimodal Large Language Models on Industrial Construction Drawing Set, Benchmark description; Table 1; Figure 2.
Evaluation Scope & Methodology
Samples, denominators, uncertainty, sources and limits
Approach
Compare models within each task and supplied-context condition; do not pool different task metrics.
Cohorts and samples
| Cohort | Sample | Definition |
|---|---|---|
| ConstructionQA supplied-context questions | 120 question–answer pairs | Approximately 150+ sheets across eight disciplines; task-specific subsets. |
Reported measures
| Measure | Unit | Cohort | Denominator | Uncertainty | Evidence |
|---|---|---|---|---|---|
| Information extraction F1-BoW, GPT-5.4 | Score from 0 to 1 | ConstructionQA supplied-context questions | Information-extraction task subset of the 120-pair benchmark | None reported | Reported in source |
| Information extraction F1-BoW, Gemini 2.5 Flash | Score from 0 to 1 | ConstructionQA supplied-context questions | Information-extraction task subset of the 120-pair benchmark | None reported | Reported in source |
| Cross-discipline coordination combined score, GPT-5.4 | Score from 0 to 1 | ConstructionQA supplied-context questions | Cross-discipline coordination task subset, Table 1 | None reported | Reported in source |
| Sheet-reference F1: GPT-5.4 | Score from 0 to 1 | ConstructionQA supplied-context questions | Same ConstructionQA sheet-reference task under distractor and guaranteed-context conditions | None reported | Reported in source |
| Sheet-reference F1: Gemini 2.5 Flash | Score from 0 to 1 | ConstructionQA supplied-context questions | Same ConstructionQA sheet-reference task under distractor and guaranteed-context conditions | None reported | Reported in source |
| Sheet-reference F1: GPT-4.1 Nano | Score from 0 to 1 | ConstructionQA supplied-context questions | Same ConstructionQA sheet-reference task under distractor and guaranteed-context conditions | None reported | Reported in source |
Limitations
- Supplied relevant sheets do not test full-corpus retrieval.
- Guaranteed context does not improve every task.
- Native-PDF conditions are separate from Table 1.
Sources
- ConstructionQA: Evaluating Multimodal Large Language Models on Industrial Construction Drawing Set, Benchmark description; Table 1; Figure 2 (Supplied manuscript reviewed November 2026). Open approved PDF