Validate multimodal interpretation of operator behavior
Planned comparison of AI interpretations against annotated behavior and computed signals.
My contribution
Design an evaluation of phase identification, behavioral interpretation, and propagation of upstream errors.
The next doctoral question
The behavioral manuscript measures differences between experience groups. The next proposed study asks whether multimodal models can correctly interpret what happened during a simulator session. The reference evidence should combine human annotations, independently checked signal calculations, and experienced-operator assessment.
Proposed evaluation
The plan considers cycle and phase identification, interpretation of joystick and gaze behavior, and error propagation into later observations. The plan reserves previously used development participants and proposes a conservative 33-participant evaluation pool before exclusions, with five participant-grouped outer folds. It is not the N=36 IMU analysis, the N=40 gaze analysis, or the separate N=119 assessment study.
Grouped outer folds should keep each participant’s recordings together. Candidate models should be compared with conventional signal-processing and machine-learning methods. Annotation readiness, synchronization, exclusion rules, and participant grouping must be established before results are reported.
Release boundary
The pipeline diagram describes a proposed study. No accuracy, annotation-completeness percentage, or achieved cross-participant performance is implied. This stage validates observations before those observations become coaching advice.
Proposed architecture
A planned design. It shows what the study would test, not a result.
Validate observations before they become advice
A planned study. Candidate model observations would be compared with human annotations, checked signal calculations and experienced-operator judgments, using participant-grouped folds.
- Proposed step, no result
Session recordingsProposed
- Input
- Simulator sessions from a planned pool of 33 participants before exclusions.
- Computation
- None.
- Output
- Multimodal recordings.
- Source
- Doctoral research plan, Paper 2
Alignment and annotation checkProposed
- Input
- Recordings and annotations.
- Computation
- Establish timing alignment, annotation readiness and exclusion rules before evaluation.
- Output
- Evaluation-ready sessions.
- Source
- Doctoral research plan, Paper 2
Human annotationsProposed
- Input
- Ready sessions.
- Computation
- Reference labels for cycles, phases and behavior.
- Output
- Reference evidence.
- Source
- Doctoral research plan, Paper 2
Checked signal calculationsProposed
- Input
- Ready sessions.
- Computation
- Independently checked joystick and gaze calculations.
- Output
- Reference evidence.
- Source
- Doctoral research plan, Paper 2
Experienced-operator reviewProposed
- Input
- Ready sessions.
- Computation
- Reference judgments from experienced operators.
- Output
- Reference evidence.
- Source
- Doctoral research plan, Paper 2
Multimodal model outputsProposed
- Input
- Ready sessions.
- Computation
- Candidate models describe what happened in the session.
- Output
- Candidate observations.
- Source
- Doctoral research plan, Paper 2
Signal-processing and ML baselinesProposed
- Input
- Ready sessions.
- Computation
- Conventional methods as comparison conditions.
- Output
- Baseline observations.
- Source
- Doctoral research plan, Paper 2
Grouped-fold comparisonProposed
- Input
- Reference evidence and candidate observations.
- Computation
- Five participant-grouped outer folds; each participant stays in one fold.
- Output
- Per-component agreement.
- Source
- Doctoral research plan, Paper 2
Component errors and propagationProposed
- Input
- Comparison results.
- Computation
- Reports component errors and how they propagate into interpretation.
- Output
- Planned analysis. No result is reported.
- Source
- Doctoral research plan, Paper 2
Proposed evaluation questions
- Can models identify work cycles and phases in a session?
- Can they interpret joystick and gaze behavior correctly?
- How do component errors propagate into later observations?
Adapted from PhD Research Summary, Asif Faisal Chowdhury, Doctoral research summary; proposed interpretation evaluation.
Planned pool of 33 participants before exclusions. This is not the N=36 IMU, N=40 gaze or N=119 assessment sample. No accuracy or completeness value is implied.
Animated pulses show conceptual signal flow only. They do not show measured execution time or a live inference run.
Evaluation Scope & Methodology
Samples, denominators, uncertainty, sources and limits
Approach
Proposed participant-grouped outer validation against annotations, computed signals, and experienced-operator reference judgments.
Cohorts and samples
| Cohort | Sample | Definition |
|---|---|---|
| Proposed interpretation pool | 33 participants | Conservative planned pool before exclusions and synchronization/annotation checks; no evaluated outcome. |
Protocol
- Keep all recordings from a participant within one outer fold.
- Establish timing alignment and annotation readiness before evaluation.
- Report component errors and their propagation into interpretation separately.
Limitations
- Planned study; no model performance or validated annotation agreement is claimed.
- The conservative 33-participant pool precedes exclusions and readiness checks.
- Timing alignment and reference annotations remain unresolved.
Sources
- PhD Research Summary, Asif Faisal Chowdhury, Doctoral research summary; proposed interpretation evaluation (Supplied manuscript reviewed November 2026). Public release pending
- Portfolio website content and coding-agent brief, Paper 2 plan; conservative participant pool and readiness notes (Supplied manuscript reviewed November 2026). Public release pending