Temporal assessment of excavator proficiency
Multi-agent assessment across 119 operators and three skill levels.
My contribution
Contributed to temporal behavior analysis and evidence-linked operator assessment reports.
Separate evidence lanes
The assessment framework combines simulator telemetry calculations, video analysis, simulator-report extraction, and report assembly. Pairwise Simultaneity Index and Bimanual Coordination Score quantify temporal control organization. Each report should preserve the difference between a computed measurement and a model-generated interpretation.
Reported group comparisons
The study covers 119 operators across three skill levels. Expert versus novice Arm–Boom simultaneity is 0.542 versus 0.125, and bimanual coordination is 0.333 versus 0.109. These are aggregate comparisons, not reconstructed participant distributions or universal thresholds.
The abstract describes three or more simultaneous controls, while other text uses three-control wording. That ambiguity blocks promoting the corresponding 35.7% versus 6.3% result as an unqualified chart. The supplied recordings and report recommendations also do not establish prospective learning gains.
System architecture
Select a step, or a block in the drawing, to see its input, computation, output and source.
Three evidence lanes, one assessment report
Video, telemetry and the simulator report are analyzed in separate lanes. Telemetry calculations produce computed measures. Report assembly writes the interpretation.
- Computed measurement
- Analysis or extraction
- Model-written interpretation
Session videoInput
- Input
- Recorded simulator session video.
- Computation
- None.
- Output
- Video for analysis.
- Source
- Excavator assessment manuscript, Figures 3-1 and 3-2
Video analysisProcessing step
- Input
- Session video.
- Computation
- Analyzes operator behavior in the recording.
- Output
- Behavior observations.
- Source
- Excavator assessment manuscript, Figures 3-1 and 3-2
Behavior observationsProcessing step
- Input
- Video analysis output.
- Computation
- None.
- Output
- Observations for the report.
- Source
- Excavator assessment manuscript, Figures 3-1 and 3-2
Simulator telemetryInput
- Input
- Control telemetry from the simulator.
- Computation
- None.
- Output
- Time-series control data.
- Source
- Excavator assessment manuscript, Figures 3-1 and 3-2
PSI and BCS calculationsDeterministic
- Input
- Control telemetry.
- Computation
- Computes the Pairwise Simultaneity Index and Bimanual Coordination Score.
- Output
- Computed temporal coordination measures.
- Source
- Excavator assessment manuscript, Figures 3-1 and 3-2
Computed measuresDeterministic
- Input
- Calculation output.
- Computation
- None.
- Output
- Measurements for the report.
- Source
- Excavator assessment manuscript, Figures 3-1 and 3-2
Simulator reportInput
- Input
- The simulator’s own session report.
- Computation
- None.
- Output
- Report document.
- Source
- Excavator assessment manuscript, Figures 3-1 and 3-2
Report extractionProcessing step
- Input
- Simulator report.
- Computation
- Extracts recorded fields from the report.
- Output
- Extracted fields.
- Source
- Excavator assessment manuscript, Figures 3-1 and 3-2
Extracted fieldsProcessing step
- Input
- Extraction output.
- Computation
- None.
- Output
- Fields for the report.
- Source
- Excavator assessment manuscript, Figures 3-1 and 3-2
Report assemblyModel
- Input
- Observations, computed measures and extracted fields.
- Computation
- Assembles a written assessment that interprets the evidence.
- Output
- Draft assessment narrative.
- Source
- Excavator assessment manuscript, Figures 3-1 and 3-2
Evidence reportOutput
- Input
- Assembled narrative and its evidence.
- Computation
- Keeps computed measurements distinct from model-written interpretation.
- Output
- Assessment report. Coaching efficacy is not established.
- Source
- Excavator assessment manuscript, Figures 3-1 and 3-2
Adapted from Multi-Agent AI Framework for Objective Assessment of Excavator Operator Proficiency Through Temporal Behavior Analysis, Abstract; Tables 4.1–4.2; Figures 3-1/3-2.
Study of 119 operators across three skill levels, separate from the wearable-sensor samples. Video evidence remains an approved-media dependency.
Animated pulses show conceptual signal flow only. They do not show measured execution time or a live inference run.
Findings
Reported results, grouped by sample. Each figure keeps its own scale and links to its source and method.
Temporal assessment sample
119 operators. Three skill levels; separate assessment/reporting study.
Arm–Boom Pairwise Simultaneity Index
Reported in sourceHighest reported value: Expert, 0.542.
Data table
| Condition | Value |
|---|---|
| Expert | 0.542 |
| Novice | 0.125 |
Denominator: Expert and novice group aggregates in the 119-operator study.
Adapted from Multi-Agent AI Framework for Objective Assessment of Excavator Operator Proficiency Through Temporal Behavior Analysis, Abstract; Tables 4.1–4.2; Figures 3-1/3-2.
Bimanual Coordination Score
Reported in sourceHighest reported value: Expert, 0.333.
Data table
| Condition | Value |
|---|---|
| Expert | 0.333 |
| Novice | 0.109 |
Denominator: Expert and novice group aggregates in the 119-operator study.
Adapted from Multi-Agent AI Framework for Objective Assessment of Excavator Operator Proficiency Through Temporal Behavior Analysis, Abstract; Tables 4.1–4.2; Figures 3-1/3-2.
Evaluation Scope & Methodology
Samples, denominators, uncertainty, sources and limits
Approach
Compare reported control-coordination aggregates across skill groups, preserving the telemetry/video/report branches.
Cohorts and samples
| Cohort | Sample | Definition |
|---|---|---|
| Temporal assessment sample | 119 operators | Three skill levels; separate assessment/reporting study. |
Reported measures
| Measure | Unit | Cohort | Denominator | Uncertainty | Evidence |
|---|---|---|---|---|---|
| Arm–Boom Pairwise Simultaneity Index | Score from 0 to 1 | Temporal assessment sample | Expert and novice group aggregates in the 119-operator study | None reported | Reported in source |
| Bimanual Coordination Score | Score from 0 to 1 | Temporal assessment sample | Expert and novice group aggregates in the 119-operator study | None reported | Reported in source |
Limitations
- This 119-operator study is separate from the N=36 classifier and N=40 gaze analysis.
- Generated assessment reports do not establish coaching efficacy.
- Three-control versus three-or-more wording requires reconciliation.
- Control-usage categories must not be stacked without resolving overlap definitions.
Sources
- Multi-Agent AI Framework for Objective Assessment of Excavator Operator Proficiency Through Temporal Behavior Analysis, Abstract; Tables 4.1–4.2; Figures 3-1/3-2 (Supplied manuscript reviewed November 2026). Open approved PDF