Lightweight Multimodal Emotion Recognition with Intermediate Fusion for Edge Deployment

DOI

10.2991/978-94-6239-828-3_16How to use a DOI?

Keywords

Multimodal Emotion Recognition; Gated Multimodal Unit; Lightweight Architecture; Intermediate Fusion; Edge Deployment

Abstract

Multimodal emotion recognition benefits affective computing applications such as companion robots, driver monitoring, and assistive healthcare, yet most audio-visual systems rely on heavy backbones and static fusion mechanisms that are unsuitable for edge devices or cannot adapt to input-dependent modality reliability. This paper proposes a lightweight framework using EfficientNetV2-B0 for facial features and a compact 1D-CNN for prosodic audio features (MFCC + Delta + Delta-Delta), integrated through a Gated Multimodal Unit (GMU) that adaptively weighs each modality via a learned gate. It targets a Raspberry Pi 4B with constraints of <12M parameters, <2 GFLOPs, <50 MB model size, and <100 ms inference time. Evaluation on RAVDESS (1,440 recordings, 8 emotion classes) uses actor-independent 5-fold group cross-validation, comparing GMU with static Concatenation and Cross-Attention fusion under identical protocols, together with masking, ablation, and roofline analyses. GMU achieves 94.17% (± 3.94%) mean cross-validation accuracy and held-out test accuracy of 77.08% (single model) and 79.17% (5-fold ensemble), with 7.75M parameters, 1.41 GFLOPs, 30.21 MB model size, and 15.22 ms inference latency, meeting all deployment constraints, unlike Cross-Attention (78.33% test accuracy, 19.69M parameters). Masking shows audio adds 9.58 percentage points over the visual-only configuration, and roofline analysis indicates a compute-bound regime consistent with Raspberry Pi 4B characteristics. GMU offers the best accuracy-efficiency trade-off among constraint-compliant fusion methods and is recommended as the default intermediate fusion strategy for real-time, resource-constrained emotion recognition; Cross-Attention may suit settings without edge constraints, at the cost of a larger memory and parameter footprint.

Copyright

© 2026 The Author(s)

Open Access

Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

Download article (PDF)

TY  - CONF
AU  - Donny Halim
AU  - Fitra Abdurrachman Bachtiar
AU  - Tirana Noor Fatyanosa
PY  - 2026
DA  - 2026/11/04
TI  - Lightweight Multimodal Emotion Recognition with Intermediate Fusion for Edge Deployment
BT  - Proceedings of the 2026 11th International Conference on Sustainable Information Engineering and Technology (SIET 2026)
PB  - Atlantis Press
SP  - 178
EP  - 187
SN  - 2589-4900
UR  - https://doi.org/10.2991/978-94-6239-828-3_16
DO  - 10.2991/978-94-6239-828-3_16
ID  - Halim2026
ER  -