AuditGuard: verifying industrial energy reports
96.2% mutant-level exact-match
96.2% exact-match with rules and a unit-aware calculator.
My contribution
Designed the report-validation workflow and evaluated deterministic and model-assisted checking.
From report claims to checkable findings
Industrial energy reports repeat quantities across calculations, tables, and narrative recommendations. AuditGuard separates number extraction from verification. Its unit-aware engine recomputes derived quantities and applies eleven checks covering consistency, report structure, terminology, and historical plausibility.
An optional language-model branch links tokens to structured claims. A direct model checker is a separate comparison path. Neither path replaces the need to identify operands, preserve units, or show a reviewer why a finding was raised.
What the comparison changed
The strongest reported condition was rules plus calculator, ahead of both the hybrid and direct 8B model. This result supports using deterministic tools for the numeric core and giving language models a bounded role. Exact-match means the entire predicted finding set matches the expected finding set for a mutated report.
EnergyAuditBench reserves 27 source reports for calibration and uses 81 held-out reports to create 1,620 mutated reports containing 3,249 seeded errors across seven classes. The 30B direct-model result and later 20B hybrid replay belong to smaller comparison slices. They remain separate evidence conditions.
Review and release
Reviewer output should connect a finding to the source quantity, recomputed value, check, and evidence. Public demonstrations require synthetic report data. The current AuditGuard draft is a separate document from both the energy-audit review and the excavator assessment paper.
System architecture
Select a step, or a block in the drawing, to see its input, computation, output and source.
Deterministic verification with a bounded model role
Numbers are extracted and recomputed by a unit-aware calculator. An optional model links tokens to claims. A direct model checker is drawn as a separate comparison path.
- Deterministic
- Model, optional
- Comparison path
- Human review
Assessment reportInput
- Input
- An industrial energy assessment report with tables, calculations and recommendations.
- Computation
- None. The report enters every path unchanged.
- Output
- Report text and tables.
- Source
- AuditGuard manuscript, Figure 1
Number extractionDeterministic
- Input
- Report text and tables.
- Computation
- Extracts quantities and their units from the report.
- Output
- Numeric operands with units.
- Source
- AuditGuard manuscript, Figure 1
LLM token linking (optional)Model
- Input
- Report tokens.
- Computation
- Links tokens to structured claims. It does not supply numeric operands.
- Output
- Claim links for the checks.
- Source
- AuditGuard manuscript, Figure 1
Unit-aware calculator and checksDeterministic
- Input
- Extracted operands, units and claim links.
- Computation
- Recomputes derived quantities and applies eleven checks: consistency, structure, terminology and historical plausibility.
- Output
- Pass or finding for each check.
- Source
- AuditGuard manuscript, Figure 1
Findings with evidenceOutput
- Input
- Check results.
- Computation
- Connects each finding to the source quantity, recomputed value and check.
- Output
- Finding set compared against the expected set (exact-match).
- Source
- AuditGuard manuscript, Figure 1
Reviewer decisionHuman review
- Input
- Findings with their evidence.
- Computation
- A person reviews why each finding was raised.
- Output
- Accepted or rejected findings.
- Source
- AuditGuard manuscript, Figure 1
Direct LLM checkerComparison path
- Input
- Report text.
- Computation
- A language model checks the report directly, bypassing extraction and the calculator.
- Output
- Model findings for the comparison condition.
- Source
- AuditGuard manuscript, Figure 1
Model findingsComparison path
- Input
- Direct checker output.
- Computation
- Scored with the same exact-match rule.
- Output
- Comparison result, reported separately.
- Source
- AuditGuard manuscript, Figure 1
Adapted from Trustworthy validation of industrial energy assessment reports: a neuro-symbolic validator and seeded-error benchmark, Abstract; Tables 4–6; Figure 1.
The model never invents numeric operands. Operands come from extraction.
Animated pulses show conceptual signal flow only. They do not show measured execution time or a live inference run.
Findings
Reported results, grouped by sample. Each figure keeps its own scale and links to its source and method.
Held-out seeded-error evaluation
1,620 mutated reports. 81 source reports; 3,249 seeded errors; excludes 27 calibration reports.
Mutant-level exact-match
Reported in sourceHighest reported value: Rules plus calculator, 96.2%.
Data table
| Condition | Value |
|---|---|
| Rules plus calculator | 96.2% |
| Hybrid 8B | 43.0% |
| Direct LLM 8B | 0.1% |
Denominator: 1,620 mutated reports in the full held-out evaluation.
Adapted from Trustworthy validation of industrial energy assessment reports: a neuro-symbolic validator and seeded-error benchmark, Abstract; Tables 4–6; Figure 1.
Minimum per-class recall
Reported in sourceMinimum per-class recall, Rules plus calculator: 0.998.
0.998Rules plus calculator
Denominator: Seven seeded-error classes on the full held-out slice.
Adapted from Trustworthy validation of industrial energy assessment reports: a neuro-symbolic validator and seeded-error benchmark, Abstract; Tables 4–6; Figure 1.
False-positive findings
Reported in sourceFalse-positive findings, Rules plus calculator: 52.
52Rules plus calculator
Denominator: Full held-out slice: 81 reports and 1,620 mutants.
Adapted from Trustworthy validation of industrial energy assessment reports: a neuro-symbolic validator and seeded-error benchmark, Abstract; Tables 4–6; Figure 1.
Evaluation Scope & Methodology
Samples, denominators, uncertainty, sources and limits
Approach
Compare exact expected finding sets on a held-out source-report slice; preserve separate calibration and model replay cohorts.
Cohorts and samples
| Cohort | Sample | Definition |
|---|---|---|
| Held-out seeded-error evaluation | 1,620 mutated reports | 81 source reports; 3,249 seeded errors; excludes 27 calibration reports. |
Reported measures
| Measure | Unit | Cohort | Denominator | Uncertainty | Evidence |
|---|---|---|---|---|---|
| Mutant-level exact-match | Percent | Held-out seeded-error evaluation | 1,620 mutated reports in the full held-out evaluation | None reported | Reported in source |
| Minimum per-class recall | Score from 0 to 1 | Held-out seeded-error evaluation | Seven seeded-error classes on the full held-out slice | None reported | Reported in source |
| False-positive findings | Count | Held-out seeded-error evaluation | Full held-out slice: 81 reports and 1,620 mutants | None reported | Reported in source |
Limitations
- Seeded errors are controlled mutations, not an observed production defect rate.
- Confidential client reports cannot be released as demonstration fixtures.
- Larger-model results use separate subsets and cannot be pooled with the full evaluation.
Sources
- Trustworthy validation of industrial energy assessment reports: a neuro-symbolic validator and seeded-error benchmark, Abstract; Tables 4–6; Figure 1 (Supplied manuscript reviewed November 2026). Public release pending