Zero-shot locomotion prediction from vision and speech
87.87% weighted F1, zero-shot GPT-4o
87.87% weighted F1 from GPT-4o without task-specific training.
My contribution
Co-authored a comparison of zero-shot and fine-tuned models for construction locomotion prediction.
Compare training conditions explicitly
Smart glasses capture field-of-view frames and speech. The study uses speech transcription and multimodal representations to compare fine-tuned CLIP, zero-shot GPT-4o, and fine-tuned ImageBind for construction locomotion modes.
Fine-tuned CLIP reaches 90.05% weighted F1, GPT-4o reaches 87.87%, and fine-tuned ImageBind reaches 78.84%. Training condition is part of each result. The comparison does not imply that one architecture replaces controller validation or generalizes to unseen participants.
Article evidence
The supplied research article prints an acceptance date of March 23, 2026 and DOI 10.26599/JIC.2026.9180128. The matching public PDF has the same title. The record uses the evidenced accepted stage while publisher metadata and reuse terms remain to be verified.
Deployment boundary
The output is a locomotion prediction from recorded multimodal evidence. No physical actuator safety, field generalization, or real-time closed-loop control guarantee follows from its F1 score. Exact confusion-matrix cells and approved images are needed for richer empirical graphics.
System architecture
Select a step, or a block in the drawing, to see its input, computation, output and source.
Three evaluated paths from vision and speech
Smart glasses capture frames and speech. Speech is transcribed. Three prediction paths are compared offline, each with its training condition.
- Capture
- Model
- Offline evaluation boundary
Smart glassesInput
- Input
- Worker wearing smart glasses during construction tasks.
- Computation
- Captures field of view and speech.
- Output
- Frames and audio.
- Source
- Zero-shot locomotion article, Figure 1
Field-of-view framesInput
- Input
- Smart-glasses video.
- Computation
- None shown here.
- Output
- Visual input.
- Source
- Zero-shot locomotion article, Figure 1
Speech audioInput
- Input
- Smart-glasses microphone.
- Computation
- None.
- Output
- Audio for transcription.
- Source
- Zero-shot locomotion article, Figure 1
Whisper transcriptionModel
- Input
- Speech audio.
- Computation
- Speech-to-text transcription.
- Output
- Transcript text.
- Source
- Zero-shot locomotion article, Figure 1
Modality inputsInput
- Input
- Frames and transcript.
- Computation
- Prepared as model inputs. The article defines which inputs each variant uses.
- Output
- Inputs for each evaluated path.
- Source
- Zero-shot locomotion article, Figure 1
CLIP, fine-tunedModel
- Input
- Modality representations, per the article variant.
- Computation
- Fine-tuned prediction path. Modality and temporal variants are evaluated separately.
- Output
- Locomotion mode.
- Source
- Zero-shot locomotion article, Figure 1
GPT-4o, zero-shotModel
- Input
- Vision and speech inputs.
- Computation
- Zero-shot prediction with no task fine-tuning.
- Output
- Locomotion mode.
- Source
- Zero-shot locomotion article, Figure 1
ImageBind, fine-tunedModel
- Input
- Modality representations, per the article variant.
- Computation
- Fine-tuned prediction path.
- Output
- Locomotion mode.
- Source
- Zero-shot locomotion article, Figure 1
Predicted modeOutput
- Input
- One prediction per evaluated path.
- Computation
- Scored offline by weighted F1 on the article evaluation split.
- Output
- Research result, not a controller certification.
- Source
- Zero-shot locomotion article, Figure 1
Adapted from Exoskeleton locomotion mode prediction in construction using GPT-4o: Zero-shot learning from vision and speech, Article first page; Figure 6; evaluation protocol.
Offline research comparison. The article describes sample splitting with oversampling after the split; participant-held-out validation is not established.
Animated pulses show conceptual signal flow only. They do not show measured execution time or a live inference run.
Findings
Reported results, grouped by sample. Each figure keeps its own scale and links to its source and method.
Article evaluation split
Study-defined held-out sample split; oversampling follows the split. Participant-held-out evaluation is not established.
Weighted F1
Reported in sourceHighest reported value: CLIP, fine-tuned, 90.05%.
Data table
| Condition | Value |
|---|---|
| CLIP, fine-tuned | 90.05% |
| GPT-4o, zero-shot | 87.87% |
| ImageBind, fine-tuned | 78.84% |
Denominator: Article evaluation samples; sample splitting does not establish participant-held-out validation.
Adapted from Exoskeleton locomotion mode prediction in construction using GPT-4o: Zero-shot learning from vision and speech, Article first page; Figure 6; evaluation protocol.
Evaluation Scope & Methodology
Samples, denominators, uncertainty, sources and limits
Approach
Compare weighted F1 within the article evaluation; retain fine-tuned versus zero-shot conditions and the source split definition.
Cohorts and samples
| Cohort | Sample | Definition |
|---|---|---|
| Article evaluation split | Not stated in a structured count | Study-defined held-out sample split; oversampling follows the split. Participant-held-out evaluation is not established. |
Reported measures
| Measure | Unit | Cohort | Denominator | Uncertainty | Evidence |
|---|---|---|---|---|---|
| Weighted F1 | Percent | Article evaluation split | Article evaluation samples; sample splitting does not establish participant-held-out validation | None reported | Reported in source |
Limitations
- The supplied article describes sample splitting and oversampling after the split.
- Participant-held-out validation has not been established.
- Offline prediction results do not certify an exoskeleton controller.
- Publisher metadata and final reuse terms need independent verification.
Sources
- Exoskeleton locomotion mode prediction in construction using GPT-4o: Zero-shot learning from vision and speech, Article first page; Figure 6; evaluation protocol (Supplied manuscript reviewed November 2026). Open approved PDF