Fusion-Based Multimodal Emotion Recognition System for Mental Health Support

DOI

10.2991/978-94-6239-789-7_28How to use a DOI?

Keywords

Computer Vision; NLP; CNN; Mental Health; RoBERTa

Abstract

The growth of online platforms has opened new opportunities to support mental health, yet many existing automated tools still fall short because they rely on only one type of input. Systems that focus solely on text or facial cues often misinterpret emotions—particularly when people mask what they truly feel through their speech or expressions. This paper introduces a novel multimodal framework designed to provide a more complete and accurate assessment of mental well-being. The framework integrates two complementary AI components: a Natural Language Processing (NLP) module powered by a RoBERTa model to explore emotional nuances in text, and a Computer Vision (CV) module using a Convolutional Neural Network (CNN) to interpret facial expressions in real time. By applying a late fusion technique to merge insights from both channels, the system can spot inconsistencies between verbal and nonverbal signals. This combined approach reduces the risk of false negatives and results in a more dependable and empathetic method for early mental health monitoring. The paper details the system’s architecture, development process, and laboratory evaluation. Results indicate that the proposed multimodal approach outperforms single-mode systems in identifying subtle emotional cues, suggesting strong potential for improving digital mental health support.

Copyright

© 2026 The Author(s)

Open Access

Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

Download article (PDF)

TY  - CONF
AU  - Shalini Singh
AU  - Shweta Kaushik
AU  - Monica Batra
PY  - 2026
DA  - 2026/10/08
TI  - Fusion-Based Multimodal Emotion Recognition System for Mental Health Support
BT  - Proceedings of theInternational Conference on Innovation and Practices in Artificial Intelligence (ICIPAI 2026)
PB  - Atlantis Press
SP  - 403
EP  - 410
SN  - 1951-6851
UR  - https://doi.org/10.2991/978-94-6239-789-7_28
DO  - 10.2991/978-94-6239-789-7_28
ID  - Singh2026
ER  -