Context-aware locomotion prediction
94.3% reported classification accuracy
94.3% reported accuracy with local perception, API reasoning, and a clarification gate.
My contribution
Integrated local scene captioning, voice input, recent locomotion history, and human clarification.
Perception and reasoning have different jobs
A local Moondream vision-language model generates scene descriptions. GPT-4o-mini combines those descriptions with speech and recent locomotion history. A confidence gate requests clarification when the prediction is uncertain. The architecture therefore includes a local perception boundary and an API reasoning boundary.
Representative evaluation
The paper describes three runs over five video-recorded scenarios. Its representative run has 35 command events across seven classes and reports 33 correct predictions, or 94.3% accuracy. Crouch walk and end crouch walk are each correct in four of five events; the other listed classes are correct in five of five.
Two of thirty-five events involve clarification. Pre-clarification predictions and corrected outputs must remain separate. Figure 3 gives median end-to-end latency of 0.49 seconds and a 95th percentile of 2.34 seconds; the prose rounds the median to approximately 0.5 seconds and the mean to approximately 0.7 seconds.
What remains unproven
These reported command events support a prototype comparison. They do not establish safe actuator timing, calibrated confidence, or effectiveness across live construction conditions. A public walkthrough needs approved recorded media, and a distribution explorer needs the event timing export.
System architecture
Select a step, or a block in the drawing, to see its input, computation, output and source.
Local perception, external reasoning, human clarification
A local model describes the scene. An external API model reasons over that description, speech and recent locomotion. A confidence gate asks the worker to clarify when uncertain.
- Model
- Gate
- Human clarification
- Local or API boundary
Video framesInput
- Input
- Recorded scenario video.
- Computation
- None.
- Output
- Frames for scene description.
- Source
- Context-aware locomotion manuscript, Figure 1
SpeechInput
- Input
- Worker speech during the scenario.
- Computation
- None shown here.
- Output
- Speech context for reasoning.
- Source
- Context-aware locomotion manuscript, Figure 1
Locomotion historyInput
- Input
- Recent locomotion modes.
- Computation
- Kept as short-term memory.
- Output
- History context for reasoning.
- Source
- Context-aware locomotion manuscript, Figure 1
Moondream scene descriptionModel
- Input
- Video frames.
- Computation
- A local vision-language model describes the scene.
- Output
- Scene description text.
- Source
- Context-aware locomotion manuscript, Figure 1
GPT-4o-mini reasoningModel
- Input
- Scene description, speech and locomotion history.
- Computation
- An external API model predicts the next locomotion command.
- Output
- Predicted mode with a confidence signal.
- Source
- Context-aware locomotion manuscript, Figure 1
Confidence gateProcessing step
- Input
- Prediction and confidence signal.
- Computation
- Routes confident predictions to output and uncertain ones to clarification.
- Output
- Predicted mode, or a clarification request.
- Source
- Context-aware locomotion manuscript, Figure 1
Predicted modeOutput
- Input
- Confident prediction.
- Computation
- None.
- Output
- Model prediction, counted before any clarification.
- Source
- Context-aware locomotion manuscript, Figure 1
Human clarificationHuman review
- Input
- Clarification request.
- Computation
- The worker answers a clarifying question.
- Output
- Corrected command.
- Source
- Context-aware locomotion manuscript, Figure 1
Clarified modeOutput
- Input
- Corrected command.
- Computation
- None.
- Output
- Clarified output, kept separate from the model prediction.
- Source
- Context-aware locomotion manuscript, Figure 1
Adapted from Multimodal LLM Framework for Context-Aware Locomotion Prediction in Construction Exoskeletons, Table 1; Figure 3 caption; evaluation text.
The reasoning step uses an external API, so the whole system is not offline.
A confidence gate over a 35-event representative run does not establish calibrated uncertainty or safe actuator timing.
Animated pulses show conceptual signal flow only. They do not show measured execution time or a live inference run.
Findings
Reported results, grouped by sample. Each figure keeps its own scale and links to its source and method.
Representative command-event run
35 command events. Seven classes with five events each; 33/35 reported correct; paper describes three runs and five scenarios.
Reported classification accuracy
Reported in sourceReported classification accuracy, Representative run: 94.3%.
Data table
| Condition | Value |
|---|---|
| Representative run | 94.3% |
Denominator: 33 correct predictions out of 35 command events in the representative run.
Adapted from Multimodal LLM Framework for Context-Aware Locomotion Prediction in Construction Exoskeletons, Table 1; Figure 3 caption; evaluation text.
Reported latency evaluation
Figure 3 caption reports latency quantiles. Event-level timing export and exact event denominator are not supplied.
Median end-to-end latency
Reported in sourceMedian end-to-end latency, Median: 0.49 s.
0.49 sMedian
Denominator: Event-level latency observations summarized in Figure 3; exact count requires export.
Adapted from Multimodal LLM Framework for Context-Aware Locomotion Prediction in Construction Exoskeletons, Table 1; Figure 3 caption; evaluation text.
95th-percentile end-to-end latency
Reported in source95th-percentile end-to-end latency, 95th percentile: 2.34 s.
2.34 s95th percentile
Denominator: Event-level latency observations summarized in Figure 3; exact count requires export.
Adapted from Multimodal LLM Framework for Context-Aware Locomotion Prediction in Construction Exoskeletons, Table 1; Figure 3 caption; evaluation text.
Evaluation Scope & Methodology
Samples, denominators, uncertainty, sources and limits
Approach
Keep the representative 35-event result separate from multi-run claims and distinguish model output from clarification-corrected output.
Cohorts and samples
| Cohort | Sample | Definition |
|---|---|---|
| Representative command-event run | 35 command events | Seven classes with five events each; 33/35 reported correct; paper describes three runs and five scenarios. |
| Reported latency evaluation | Not stated in a structured count | Figure 3 caption reports latency quantiles. Event-level timing export and exact event denominator are not supplied. |
Reported measures
| Measure | Unit | Cohort | Denominator | Uncertainty | Evidence |
|---|---|---|---|---|---|
| Reported classification accuracy | Percent | Representative command-event run | 33 correct predictions out of 35 command events in the representative run | None reported | Reported in source |
| Median end-to-end latency | Seconds | Reported latency evaluation | Event-level latency observations summarized in Figure 3; exact count requires export | None reported | Reported in source |
| 95th-percentile end-to-end latency | Seconds | Reported latency evaluation | Event-level latency observations summarized in Figure 3; exact count requires export | None reported | Reported in source |
Limitations
- Simulated live evaluation from recorded scenarios, not a field safety trial.
- The reasoning step uses an external API; the whole system is not offline.
- A 35-event representative run does not establish calibrated uncertainty.
- Event-level timings are needed to reconstruct a latency distribution.
Sources
- Multimodal LLM Framework for Context-Aware Locomotion Prediction in Construction Exoskeletons, Table 1; Figure 3 caption; evaluation text (Supplied manuscript reviewed November 2026). Open approved PDF