Energy-Aware Modality Selection for Wearable Multimodal Activity Recognition

This thesis focuses on developing an energy-efficient multimodal framework for human activity understanding. Advanced wearable devices can capture rich multimodal signals, including video, audio, and IMU data. However, redundancy across modalities and limited energy budgets make it impractical to analyze all modalities continuously. To address this challenge, this research first builds unimodal models for audio, video, and IMU data using CNN- and Transformer-based architectures, while estimating each modality's energy cost based on computation and sensor power consumption. The second stage uses a Predict-and-Select framework. At each time step, a prediction head classifies the current activity, while a policy scheduler selects the next subset of modalities under the energy budget. This work contributes a unified framework for joint activity classification and energy-aware modality scheduling, enabling cost-effective activity understanding for scalable deployment in practical systems.