Measuring excavator operator behavior
94.5% mean balanced accuracy
94.5% mean balanced accuracy from wearable joystick signals.
My contribution
Analyzed joystick-mounted inertial measurements and separate gaze measures to study experience-related behavior.
Measure behavior from recorded signals
Joystick-mounted inertial measurement units (IMUs) capture control motion. The classification analysis uses 36 usable trials, with 19 experienced and 17 inexperienced operators. A separate gaze analysis uses 40 participants. These branches answer complementary questions and retain separate sample definitions.
Classification and coordination
Logistic regression reports 94.5% mean balanced accuracy with a fold standard deviation of 9.0 percentage points. The protocol repeats stratified five-fold cross-validation twenty times and refits preprocessing inside each training fold. Leave-one-out performance is a separate condition.
Experienced operators show 43.2% simultaneous joystick movement versus 20.4% for inexperienced operators, measured as a share of active time. This descriptive contrast is not the classifier’s score and does not prescribe an ideal overlap percentage for training.
Visual behavior
Eight of sixteen eligible gaze/head metrics differ after correction. Pupil variability has reported group medians of 0.153 and 0.216 mm, with Cliff’s delta −0.88 and a 95% confidence interval from −1.00 to −0.69. These are aggregate associations. Participant trajectories and distributions cannot be reconstructed from them.
System architecture
Select a step, or a block in the drawing, to see its input, computation, output and source.
Two sensor branches with separate samples
Joystick IMUs feed classification and coordination analysis. Eye tracking feeds a separate gaze comparison. The branches use different samples and are not combined into one classifier.
- Signal processing and statistics
- Classifier
- Reported result
Simulator sessionInput
- Input
- One excavator simulator task.
- Computation
- None.
- Output
- Recorded sensor streams.
- Source
- Wearable-sensor manuscript, Sections 3 and 4
Joystick IMUsInput
- Input
- Joystick-mounted inertial measurement units.
- Computation
- Records control motion.
- Output
- IMU signals from 36 usable trials.
- Source
- Wearable-sensor manuscript, Sections 3 and 4
Motion featuresDeterministic
- Input
- IMU signals.
- Computation
- Feature extraction; preprocessing is refitted inside each training fold.
- Output
- Per-trial features.
- Source
- Wearable-sensor manuscript, Sections 3 and 4
Four classifiersModel
- Input
- Per-trial features.
- Computation
- Logistic regression, LDA, radial-basis SVM and random forest under twenty repeats of stratified five-fold cross-validation.
- Output
- Experienced or inexperienced label.
- Source
- Wearable-sensor manuscript, Sections 3 and 4
Experience classOutput
- Input
- Classifier predictions.
- Computation
- Mean balanced accuracy with fold SD.
- Output
- Classification result.
- Source
- Wearable-sensor manuscript, Sections 3 and 4
Coordination analysisDeterministic
- Input
- IMU activity.
- Computation
- Both-active time as a share of active time.
- Output
- Group mean simultaneous movement.
- Source
- Wearable-sensor manuscript, Sections 3 and 4
Simultaneous movementOutput
- Input
- Coordination measure.
- Computation
- Descriptive group comparison.
- Output
- Not a training target and not the classifier score.
- Source
- Wearable-sensor manuscript, Sections 3 and 4
Eye tracker, gaze and headInput
- Input
- Eye tracking during the task.
- Computation
- Records gaze and head movement.
- Output
- Signals from 40 participants.
- Source
- Wearable-sensor manuscript, Sections 3 and 4
Eligible gaze metricsDeterministic
- Input
- Gaze and head signals.
- Computation
- Computes sixteen eligible gaze and head metrics.
- Output
- Per-participant metrics.
- Source
- Wearable-sensor manuscript, Sections 3 and 4
Corrected comparisonsDeterministic
- Input
- Per-participant metrics.
- Computation
- Group comparisons with Cliff's delta and corrected q values.
- Output
- Effect sizes with 95% CIs.
- Source
- Wearable-sensor manuscript, Sections 3 and 4
Group differencesOutput
- Input
- Corrected comparisons.
- Computation
- None.
- Output
- Associations only; pupil variation alone does not establish cognitive load.
- Source
- Wearable-sensor manuscript, Sections 3 and 4
Adapted from Skill Classification and Behavioral Analysis of Excavator Operators Using Wearable Sensors, Table 4; Section 4.2.1; Figure 6; Table 5.
No combined IMU and gaze classifier result is reported.
Animated pulses show conceptual signal flow only. They do not show measured execution time or a live inference run.
Findings
Reported results, grouped by sample. Each figure keeps its own scale and links to its source and method.
Usable IMU trials
36 trials. 19 experienced, 17 inexperienced; wearable classification and coordination branch.
Mean balanced accuracy, repeated five-fold validation
Reported in sourceHighest reported value: Logistic regression, 94.5%. Whiskers show one standard deviation, not a confidence interval.
A whisker reaches beyond the scale and is drawn to the axis edge without an end cap. The data table gives the exact value.
Data table
| Condition | Value | Uncertainty |
|---|---|---|
| Logistic regression | 94.5% | SD 9.0 percentage points |
| Linear discriminant analysis | 92.5% | SD 11.6 percentage points |
| Radial-basis SVM | 92.2% | SD 10.7 percentage points |
| Random forest | 89.0% | SD 12.2 percentage points |
Denominator: Usable 36-trial IMU sample; twenty repeats of stratified five-fold cross-validation.
Adapted from Skill Classification and Behavioral Analysis of Excavator Operators Using Wearable Sensors, Table 4; Section 4.2.1; Figure 6; Table 5.
Mean simultaneous joystick movement
Reported in sourceHighest reported value: Experienced, 43.2%.
Data table
| Condition | Value |
|---|---|
| Experienced | 43.2% |
| Inexperienced | 20.4% |
Denominator: Both-active time as a share of active time; reported group means.
Adapted from Skill Classification and Behavioral Analysis of Excavator Operators Using Wearable Sensors, Table 4; Section 4.2.1; Figure 6; Table 5.
Separate gaze analysis
40 participants. Gaze/head comparisons; not the IMU classifier cohort.
Median pupil variability
Reported in sourceHighest reported value: Inexperienced, 0.216 mm.
Data table
| Condition | Value |
|---|---|
| Experienced | 0.153 mm |
| Inexperienced | 0.216 mm |
Denominator: Separate gaze analysis; reported experience-group medians.
Adapted from Skill Classification and Behavioral Analysis of Excavator Operators Using Wearable Sensors, Table 4; Section 4.2.1; Figure 6; Table 5.
Cliff's delta: experienced versus inexperienced
Reported in sourceAll 8 reported effects are negative, from −0.88 to −0.44, and every 95% CI excludes zero.
Data table
| Condition | Value | Uncertainty | Corrected significance |
|---|---|---|---|
| Pupil variability | −0.88 | 95% CI [−1.00, −0.69] | q < 0.0001 |
| Mean saccade amplitude | −0.87 | 95% CI [−1.00, −0.65] | q < 0.0001 |
| Vertical gaze spread | −0.78 | 95% CI [−0.96, −0.55] | q = 0.0001 |
| Mean saccade duration | −0.69 | 95% CI [−0.94, −0.37] | q = 0.0007 |
| Stationary gaze entropy | −0.69 | 95% CI [−0.91, −0.41] | q = 0.0007 |
| Head rotation rate | −0.66 | 95% CI [−0.89, −0.38] | q = 0.001 |
| Median fixation duration | −0.49 | 95% CI [−0.76, −0.17] | q = 0.0177 |
| 2D gaze dispersion | −0.44 | 95% CI [−0.73, −0.10] | q = 0.0359 |
Denominator: 40 participants, 20 per group; q values Benjamini-Hochberg corrected across all 16 eligible metrics.
Adapted from Skill Classification and Behavioral Analysis of Excavator Operators Using Wearable Sensors, Table 4; Section 4.2.1; Figure 6; Table 5.
Evaluation Scope & Methodology
Samples, denominators, uncertainty, sources and limits
Approach
Repeated stratified five-fold cross-validation with twenty repeats and training-fold preprocessing; separate corrected gaze comparisons.
Cohorts and samples
| Cohort | Sample | Definition |
|---|---|---|
| Usable IMU trials | 36 trials | 19 experienced, 17 inexperienced; wearable classification and coordination branch. |
| Separate gaze analysis | 40 participants | Gaze/head comparisons; not the IMU classifier cohort. |
Reported measures
| Measure | Unit | Cohort | Denominator | Uncertainty | Evidence |
|---|---|---|---|---|---|
| Mean balanced accuracy, repeated five-fold validation | Percent | Usable IMU trials | Usable 36-trial IMU sample; twenty repeats of stratified five-fold cross-validation | Logistic regression: SD 9.0 percentage points; Linear discriminant analysis: SD 11.6 percentage points; Radial-basis SVM: SD 10.7 percentage points; Random forest: SD 12.2 percentage points | Reported in source |
| Mean simultaneous joystick movement | Percent | Usable IMU trials | Both-active time as a share of active time; reported group means | None reported | Reported in source |
| Median pupil variability | Millimetres | Separate gaze analysis | Separate gaze analysis; reported experience-group medians | None reported | Reported in source |
| Cliff's delta: experienced versus inexperienced | Cliff's delta from −1 to 1 | Separate gaze analysis | 40 participants, 20 per group; q values Benjamini-Hochberg corrected across all 16 eligible metrics | Pupil variability: 95% CI [−1.00, −0.69]; Mean saccade amplitude: 95% CI [−1.00, −0.65]; Vertical gaze spread: 95% CI [−0.96, −0.55]; Mean saccade duration: 95% CI [−0.94, −0.37]; Stationary gaze entropy: 95% CI [−0.91, −0.41]; Head rotation rate: 95% CI [−0.89, −0.38]; Median fixation duration: 95% CI [−0.76, −0.17]; 2D gaze dispersion: 95% CI [−0.73, −0.10] | Reported in source |
Uncertainty definitions
- SD. Standard deviation as reported by the source. It describes spread across evaluation folds or runs. It is not a confidence interval.
- 95% CI. 95% confidence interval as reported by the source.
Protocol
- Keep the N=36 IMU and N=40 gaze branches separate.
- Classifier uncertainty is fold standard deviation, not a confidence interval.
- Leave-one-out accuracy belongs to a separate validation condition.
Limitations
- One simulator and task with a small sample.
- The gaze sample is separate from the usable IMU sample.
- No combined IMU–gaze classifier result or independent field validation is established.
- Descriptive group differences are not validated training targets.
- Pupil variability alone does not establish cognitive load.
Sources
- Skill Classification and Behavioral Analysis of Excavator Operators Using Wearable Sensors, Table 4; Section 4.2.1; Figure 6; Table 5 (Supplied manuscript reviewed November 2026). Public release pending