Real-Time Multimodal AI for Medical Intervention Understanding

Document Type

Article

Publication Date

2025

Abstract

Accurate, timely, and comprehensive documentation of medical interventions under battlefield conditions is critical but challenging due to operational constraints and the limitations of manual record-keeping. This paper introduces a multi-modal artificial intelligence (AI) system designed to automate the detection, verification, and documentation of tourniquet applications by integrating synchronized video and audio data streams. The visual pipeline utilizes fine-tuned YOLO11 models for real-time object detection and pose estimation, achieving precise identification and anatomical localization of tourniquets. Concurrently, an audio processing pipeline employs a Whisper speech recognition model, tailored to reliably capture verbal confirmations from medical personnel, even amid significant operational noise. A lightweight fusion algorithm harmonizes the output of these independent modalities, generating structured, time-stamped records that detail device type, anatomical positioning, and procedural context. This multimodal documentation facilitates immediate operational decision making and supports retrospective analyses. In addition, the system features an intuitive graphical user interface (GUI) that offers real-time visualization of detections and transcriptions, improving situational awareness, and minimizing cognitive load for medical personnel. The current implementation operates effectively on standard laptop hardware, with architectural considerations explicitly oriented towards future deployment on resource-constrained edge devices, such as the Raspberry Pi 5 and NVIDIA Jetson Nano. The presented results demonstrate the practicality of deploying lightweight, scalable AI-driven documentation systems in austere, tactical environments, establishing a solid foundation for extending real-time monitoring capabilities across a broader spectrum of battlefield trauma interventions. © 2025 IEEE.

Link to Full Text

Share

COinS