Enhancing emotion recognition through three modalities fusion and contrastive learning with the Mamba architecture

Abstract

Multimodal emotion recognition extracts emotional information from sequential multimodal data and classifies emotion tendencies. Current multimodal fusion methods based on artificial intelligence mainly rely on Transformers to extract features and integrate different data types. Despite their strength in learning global information, Transformers face challenges due to their quadratic complexity. Recent advances in state space models, especially the Mamba architecture, provide a promising solution by achieving global awareness with linear complexity. However, the potential of Mamba for information fusion in multimodal domains remains largely untapped. This paper introduces an innovative and efficient multimodal fusion and contrastive learning method called Fusion Mamba and Contrastive Learning, which leverages artificial intelligence for implementation and application in emotion recognition tasks. To effectively extract distinct features, the unimodal Mamba architecture is used to enhance unimodal representations. For comprehensive information fusion, the Mamba block is extended to handle dual inputs, forming a novel module called the Fusion Mamba block. This forms the basis for an architecture that incorporates three different modalities and three branches. Additionally, contrastive learning and interaction-level auxiliary classification constraints are jointly optimized to boost performance. The effectiveness of our approach, which highlights the application of artificial intelligence, is validated through experiments on three public datasets. Both quantitative and qualitative evaluations show that our method achieves state-of-the-art performance with 32.2% faster inference. Extensive ablation studies further confirm the effectiveness of the Mamba architecture in multimodal tasks.

Introduction

In recent times, the significance of human–computer interaction has grown, leading to extensive research in emotion recognition [1]. Given the variety of information, multimodal data (such as human utterances with linguistic, auditory, and visual elements) have become key resources for systems to comprehend human emotions [2]. This research area primarily aims at improving and integrating sequential emotional data from different modalities. Earlier studies mainly focused on creating sophisticated models to extract and merge emotional features from three distinct types of data (linguistic, acoustic, and visual) [3]. Lately, some attention-based [4, 5] and Transformer-based methods [6, 7] have been introduced to handle modality interactions and extract multimodal features. These methods have demonstrated improved performance compared to traditional frameworks such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs). While models based on Transformers or their combinations with other architectures excel in global modeling, they incur significant computational costs due to the quadratic growth in resource requirements caused by the self-attention mechanism’s token count [8].

Recent advancements in state space models (SSMs), especially Mamba [9], offer an effective solution by achieving global awareness with linear complexity. Mamba has demonstrated its efficacy in tasks that need long-term dependency modeling, such as natural language processing, due to its adaptable input and global information modeling capabilities. This approach maintains linear complexity, reducing computational costs and enhancing inference speed [10]. Recently, various Mamba variants have also achieved promising outcomes in computer vision tasks, including image classification [11] and medical image fusion [12]. Simultaneously, in the visual domain, some multimodal image fusion based on the Mamba architecture has also achieved success [13].

Inspired by Mamba’s success in different modalities [14, 15], we were motivated to apply Mamba to the emotion recognition task across tri-modal visual, textual, and auditory data. At present, there is limited research on the potential of SSM to integrate these three modalities, which is vital for the multimodal field.

Therefore, we propose a MER model based on Mamba. It consists of three stages: Intra-Mamba feature extraction, Inter-Mamba modality interaction and fusion, and a tri-modal contrastive learning module. The Intra-Mamba feature extraction stage comprises convolutional layers and multiple stacked Mamba blocks, leveraging CNN’s excellent early-stage processing capabilities for visual tasks and Mamba’s efficiency in extracting long-distance features. In the Inter-Mamba modality interaction and fusion stage, inspired by previous cross-attention mechanisms, we design a dual-modality Mamba fusion module (FM). Since FM is designed for interactions between two modalities. We suggest a parallel structure with three branches (TFM) for three modality pairs: ta, tv, and av, where t, a, and v stand for textual, auditory, and visual modalities, respectively. This parallel structure allows our method to extract inter-modal features while considering all relevant modalities. Finally, we introduce an unsupervised multimodal contrastive learning method to ensure that each parallel TFM is effectively trained.

The contribution of this work can be summarized as follows:

• To the best of our knowledge, our approach is among the pioneering works utilizing Mamba for tri-modal emotion recognition, demonstrating efficiency and effectiveness compared to traditional Transformers;

• We designed a parallel structure incorporating a tri-branch Mamba fusion module to effectively integrate information from different modalities. This parallel structure allows for unified modality interactions while extracting inter-modal features;

• We introduced a multimodal contrastive learning module to alleviate the modality gap in the outputs of the parallel structure. This module leverages contrastive learning-based modality-invariant features;

• We demonstrated our approach’s effectiveness using three public datasets, achieving sota performance in multimodal emotion recognition (MER). Furthermore, extensive ablation studies affirm the Mamba architecture’s efficacy in multimodal tasks.

Related work

In this section, we briefly review previous related works about multimodal emotion recognition, state space model, and contrastive learning.

Multimodal emotion recognition

MER aims to infer human emotions by analyzing information from three modalities present in videos: language, visual, and auditory. There are three main approaches to MER: fusion-based methods [16,17,18], decoupling-based methods [19,20,21,22], and cross-modal attention methods [23,24,25].

Fusion-based methods develop advanced strategies for multimodal integration. Recent approaches highlight that integrating different modalities is crucial in MER. One effective method is the unrestricted exchange of information between multiple modalities. Zadeh et al. [17] introduced the tensor fusion network (TFN), which progressively aggregates unimodal, bimodal, and trimodal interactions. Zadeh et al. [18] proposed the memory fusion network (MFN), using temporal information for modality interactions in a long-short term memory (LSTM) manner. Liu et al. [16] developed the low-rank multimodal fusion (LMF) method, using weighted low-rank matrix factorization to reduce excessive parameters in fusion models. However, multimodal feature fusion often faces challenges due to inherent heterogeneity and redundant information between different modalities.

An effective way to address the inherent heterogeneity and redundancy between different modalities is through feature decoupling or disentanglement. Decoupling-based strategies promote more effective fusion of multimodal representations. DMD [19] deconstructed the representation of each modality into two parts in an autoregressive manner, namely the modalityirrelevant/-exclusive spaces, reducing redundant information and thereby improving the effectiveness of multimodal fusion. Building on this, Weichen Dai et al. [22] proposed the multimodal information disentanglement (MInD) method to tackle the underutilization of heterogeneous modality information. The primary strategy involves decomposing the features of each modality into three distinct components through information optimization, which together provide a comprehensive view of the given input.

As Transformer networks [26] become more prevalent in natural language processing and computer vision, researchers have utilized specific self-attention mechanisms to identify correlations between different modalities [23]. Transformers’ self-attention mechanism captures more global information than traditional attention methods. This mechanism inherently generates attention maps from two independent streams (queries (Q) and keys (K) in Transformer structures), naturally applying to multimodal data processing. Consequently, most Transformer-based multimodal methods use Q and K from two modalities to compute a joint attention map for information exchange. This is known as inter-modal attention. MulT [27] introduced the Multimodal Transformer, focusing on interactions between multimodal sequences across different time steps and aligning streams from one modality to another.

While Transformer-based networks have inherent advantages for multimodal processing, their quadratic time complexity and computational resource demands associated with self-attention mechanisms make them inefficient for multimodal tasks and less scalable for tri-modal data. Specifically, cross-attention mechanisms can compute correlations for at most two modalities simultaneously. In this study, to utilize all modalities, we created a parallel structure with three branches, each designed to transfer information between two modalities.

State space model

The SSM concept was first presented in the S4 [28] model, offering a distinctive architecture that effectively models global information, surpassing traditional convolutional neural networks and Transformers. Building on S4, the S5 [29] model reduced the complexity to a linear level, and H3 [30] applied it to language modeling tasks. Mamba enhanced the SSM with an input activation mechanism, resulting in faster inference and better performance compared to similarly sized Transformers. Initially showing great promise in NLP tasks, Mamba later attracted significant attention in computer vision. Notable contributions include Vision Mamba [31], VMamba [32], and Pan-Mamba [33].

Recently, some research has extended Mamba to the visual multimodal domain. FusionMamba [8] integrated Mamba blocks into two U-shaped networks: Spatial U-Net and Spectral U-Net, achieving efficient image fusion. CFMW [34] proposed a cross-modal Mamba with weather removal capability, enhancing detection accuracy under adverse weather conditions. Audio Mamba [35] employed a self-attention-free, fully SSM-based audio classification model, achieving comparable or better performance compared to Transformers. Coupled Mamba [36] applies Mamba modules to multimodal emotion analysis by integrating visual, audio, and textual features. This method adopts a direct multimodal fusion strategy, ensuring tight information integration. However, during the fusion process, differences between modalities may affect the overall fusion performance. To address this, we propose a phased interaction strategy, where pairwise modality fusion (V-T, V-A, T-A) is first conducted to better align information from different modalities, followed by a higher-level three-modal fusion to enhance the balance and effectiveness of multimodal integration.

Contrastive learning

Contrastive learning aims to bring similar instances closer and push dissimilar ones apart in the feature space. In recent years, contrastive learning has shown remarkable success in both computer vision and natural language processing fields. A prominent example is CLIP [37] (Contrastive Language-Image Pre-training), which trains visual and textual representations jointly to achieve superior multimodal understanding. Another example is LanguageBind [38], which employs contrastive learning to align language representations with emotion information, achieving significant improvements.

Akbari et al. [39] proposed a framework for cross-modal contrastive learning, using a transformer-based architecture to enhance the learning of emotion features across multiple modalities. Inspired by the prior advances in multimodal contrastive learning, this paper employs contrastive learning as a key component to enhance our MER model.

Therefore, leveraging previous developments in multimodal contrastive learning, this paper adopts contrastive learning to provide robust feature alignment for MER.

Methodology

Preliminaries

State space model

The SSM is a continuous system that converts a 1D input $x(t) \in \mathbb {R}$ into an output $y(t) \in \mathbb {R}$ through an intermediate hidden state $h(t) \in \mathbb {R}^N$. This can be mathematically described using ordinary differential equations (ODEs):

$$\begin{aligned} \begin{aligned}&h^{\prime }(t)=\textbf{A}h(t)+\textbf{B}x(t),\\ &y(t)=\textbf{C}h(t).\end{aligned} \end{aligned}$$

(1)

In these equations, $A \in \mathbb {R}^{N \times N}$ is the evolution parameter, while $B \in \mathbb {R}^{N \times 1}$ and $C \in \mathbb {R}^{1 \times N}$ are projection parameters. This indicates that SSM demonstrates global awareness, as the current output depends on all prior input data.

When $A, B,$ and $C$ are constants, Eq. (1) defines a linear time-invariant (LTI) system, as mentioned in S4 [28]. If they vary, the system becomes a linear time-varying (LTV) system, as described in Mamba [9]. LTI systems cannot inherently perceive input content, whereas LTV systems with input awareness are designed to do so. This significant distinction allows Mamba to overcome SSM limitations.

Discretization

For SSM to be applied in deep learning (DL), it must be discretized. A timescale parameter $\Delta \in \mathbb {R}$ is introduced to convert the continuous parameters $A$ and $B$ into their discrete counterparts, denoted as $A$ and $B$. Using zero-order hold (ZOH) as the transformation method, the discrete parameters are expressed as follows:

$$\begin{aligned} \begin{aligned}&\bar{\textbf{A}}=\exp {(\Delta \textbf{A})},\\ &\bar{\textbf{B}}=(\Delta \textbf{A})^{-1}(\exp {(\Delta \textbf{A})}-\textbf{I})\cdot \Delta \textbf{B}\approx \Delta \textbf{B}. \end{aligned} \end{aligned}$$

(2)

Thus, the discrete form of Eq. (1) becomes:

$$\begin{aligned} \begin{aligned}&h_{t}=\bar{\textbf{A}}h_{t-1}+\bar{\textbf{B}}x_{t},\\ &y_{t}=\textrm{C}h_{t}. \end{aligned} \end{aligned}$$

(3)

In practice, $x_t$ is a feature vector of size $C$, and Eq. (3) operates independently on each feature.

Selective scan

In Mamba, the parameters vary with input, making it impossible to reformulate Eq. (3) into a convolutional form, thus hindering SSM parallelization. Mamba addresses this with a selective scan mechanism, integrating three classical techniques: kernel fusion, parallel scan, and recomputation. This mechanism allows Mamba to achieve high processing speed with relatively low memory usage.

Overview

The processed embeddings of the three modalities are sequentially fed into the Intra-Mamba and Tri-modal Mamba Fusion modules, where the Tri-modal Mamba Fusion module is primarily structured around the Fusion Mamba Block. It employs a hierarchical fusion strategy, first enabling direct modality interaction in the middle fusion stage, followed by further integration of the fused representations in the late fusion stage to enhance cross-modal understanding. Subsequently, sample-based tri-modal contrastive learning and interaction-level auxiliary classification constraints applied to cross-modal representations are employed. Finally, the fused features are passed to the classifier to predict emotion tendencies. The specific process is shown in Fig. 1. To improve readability, Table 1 summarizes the main symbols and tensor shapes used in our framework.

Fig. 1

The overall structure of our proposed model. The processed embeddings of the three modalities are sequentially fed into the intra-Mamba and Tri-modal Mamba Fusion modules to extract multimodal features. Subsequently, a sample-based tri-modal contrastive learning and interaction-level auxiliary classification constraints are applied. The fused features are then passed to the classifier to predict emotion tendencies

Table 1 Main symbols and tensor shapes in our framework

Modality encoder

The modality encoder aims to obtain modality-enhanced sequential representations. It mainly consists of two parts: Temporal Convolution and Intra-Mamba.

(1) Temporal convolution. To ensure that three unimodal sequence representations lie in the same space, we feed them into a 1D convolutional layer:

$$\begin{aligned} \begin{aligned} F_{a} =1\textbf{D}_{a}(F_{a}^{cova};\theta _{a}^{1D}), \\ F_{v} =\textbf{1D}_{v}(F_{v}^{facet};\theta _{v}^{1D}), \\ F_{t} =1\textbf{D}_{v}(F_{t}^{bert};\theta _{t}^{1D}), \end{aligned}\end{aligned}$$

(4)

where $\theta ^{1D}_{m}$ denotes the parameters of a one-dimensional convolutional network.

(2) Intra-modal Mamba. We introduce the intra-modal encoder to further extract unimodal features, reducing complexity compared to traditional Transformers, as shown in Fig. 2. Specifically, the Intra-modal Mamba employs a standard Mamba block structure. Here’s a detailed explanation: The input feature sequence $\textbf{H}_m \in \mathbb {R}^{B \times N \times d}$ is first layer-normalized to obtain $\textbf{H}'_m$. Then, in two independent branches, two multilayer perceptrons (MLP) project $\textbf{H}'_m$ to x and z. In the first branch, z is activated by a SiLU function, acting as a gating factor for f, resulting in $f'$. Finally, $f'$ passes through an MLP layer and a residual connection, yielding the output $\textbf{H}_{m \rightarrow m}$ for higher-level feature extraction. The detailed process is illustrated in Algorithm 1. We take $\textbf{H}_m$ as input:

$$\begin{aligned} \textbf{H}_{m\rightarrow m}=\mathrm {Intra-Mamba}(\textbf{H}_{m})\in \mathbb {R}^{B\times N\times d} \end{aligned}$$

(5)

where $\textbf{H}_m \in \mathbb {R}^{B\times N \times d}$, $\textbf{H}_{m \rightarrow m} \in \mathbb {R}^{B\times N \times d}$, $m \in \{t, a, v\}$.

Algorithm 1

Intra-Mamba

Tri-modal fusion Mamba

Currently, the Mamba architecture seldom handles multimodal information due to the lack of mechanisms similar to cross-attention. Although some work has made progress in bimodal integration, there is still a gap in tri-modal fusion. Therefore, we have designed a Fusion Mamba (FM) block. The structure is illustrated in Fig. 3. To fully integrate information from each modality, we have designed a tri-modal fusion module (TFM) consisting of three parallel bimodal Mamba fusion modules. The computation equations are defined as follows:

$$\begin{aligned} \begin{aligned} \textbf{H}_{vt}&=\textrm{FM}\left( \textbf{H}_{v\rightarrow v},\textbf{H}_{t\rightarrow t}\right) \in \mathbb {R}^{B\times N\times 2d},\\ \textbf{H}_{va}&=\textrm{FM}\left( \textbf{H}_{v\rightarrow v},\textbf{H}_{a\rightarrow a}\right) \in \mathbb {R}^{B\times N\times 2d},\\ \textbf{H}_{at}&=\textrm{FM}\left( \textbf{H}_{a\rightarrow a},\textbf{H}_{t\rightarrow t}\right) \in \mathbb {R}^{B\times N\times 2d}. \end{aligned} \end{aligned}$$

(6)

$$\begin{aligned} \textbf{H}_{TFM}=\textrm{cat}[\textbf{H}_{vt},\textbf{H}_{va},\textbf{H}_{at}] \end{aligned}$$

(7)

The inputs to FM are the different modality embeddings $\textbf{H}_{m \rightarrow m}$ obtained from the Modality Encoder. First, each modality undergoes a simple shallow fusion to obtain fused embeddings. Then, each modality fusion module constructs three branches, with each branch receiving features from two modalities. Similarly, these branches are layer-normalized, convolved, activated by SiLU, and passed through SSM to obtain outputs. Finally, the outputs are concatenated to obtain the final fused features. For more details, refer to Algorithm 2.

Algorithm 2

Fusion Mamba Block

Fig. 2

Intra-Mamba architecture

Fig. 3

Fusion Mamba architecture

Contrastive learning

To ensure the effective execution of multimodal Mamba, we integrated a contrastive learning component following the three FM modules. First, we map $\textbf{H}_{vt}$, $\textbf{H}_{va}$, and $\textbf{H}_{at}$ to a shared space, obtaining $\hat{\textit{H}}^m$. The calculation is defined as follows:

$$\begin{aligned} \widehat{H}^{m}=\textrm{MLP}(\textbf{H}_{mn}) \end{aligned}$$

(8)

where $m \in \{a, v, t\}$, $n \in \{a, v, t\}$, and $\textrm{MLP} (*)$ is a linear layer.

Next, we construct positive and negative pairs for contrastive learning. Specifically, we build sample pairs from features of different fused modalities, such as text-audio, text-visual, and audio-visual pairs. Positive sample pairs consist of corresponding features from the same sample, while negative pairs consist of any two features from unmatched samples. Assuming a training batch contains two samples $\{S_A, S_B\}$, $\hat{H}^a_A$, $\hat{H}^v_A$, and $\hat{H}^t_A$ form positive pairs with each other. Additionally, any features among $\hat{H}^a_A$, $\hat{H}^v_A$, or $\hat{H}^t_A$ and $\hat{H}^a_B$, $\hat{H}^v_B$, $\hat{H}^t_B$ from sample B constitute negative pairs.

Referring to previous works on contrastive learning [40], we construct the following loss function for contrastive learning on X and Y pairs:

$$\begin{aligned} \mathcal {L}_{CL}(X,Y)=-\log \frac{\exp (\sin (X_{i},Y_{i}))}{\sum _{j=1}^{N} \textbf{1}_{[j\ne i]}\exp \left( \sin (X_{i},Y_{j})\right) }, \end{aligned}$$

(9)

where $(X, Y) \in ((\hat{H}^a, \hat{H}^t), (\hat{H}^v, \hat{H}^t), (\hat{H}^a, \hat{H}^v))$, and X is the anchor point, $X_i, Y_i$ denote the positive pair, the $X_i, Y_j$ denote negative pairs, N is the sample size, and sim is the cosine similarity.

Furthermore, to perform more comprehensive contrastive learning, we swap the positions of X and Y, and recompute the loss $L_{CL}(Y, X)$. Thus, the final loss for contrastive learning is defined as:

$$\begin{aligned} \mathcal {L}_{CLT}(X,Y)=\frac{1}{2}(\mathcal {L}_{CL}(X,Y)+\mathcal {L}_{CL}(Y,X)). \end{aligned}$$

(10)

Finally, we improve the performance of the Tri-Modal Fusion Mamba by minimizing this loss.

Classifier

Given the cross-modal interaction representations $\textbf{H}_{m \rightarrow n}$ generated by the multimodal fusion Mamba, we perform adaptive feature fusion to obtain the final multimodal representation $\textbf{H}_{\textit{TFM}}$. Specifically, $\textbf{H}_{m \rightarrow n}$ denotes the intermediate representation after information from modality m is injected into modality n.

Table 2 The compositions of the number of samples in the CMU-MOSI, CMU-MOSEI, and MELD datasets

Table 3 Performance comparison on CMU-MOSI and CMU-MOSEI datasets

The fused representation is projected to the prediction space as:

$$\begin{aligned} \textbf{Y} = \textrm{MLP}(\textbf{H}_{\textit{TFM}}), \end{aligned}$$

(11)

where $\textrm{MLP}$ denotes a two-layer fully connected network.

In addition to the main classification head, we introduce an interaction-level auxiliary classifier for each cross-modal representation:

$$\begin{aligned} \textbf{Y}_{mn} = \textrm{MLP}(\textbf{H}_{m \rightarrow n}), \end{aligned}$$

(12)

where $m,n \in \{a, v, t\}$ and $m \ne n$.

Based on these predictions, the classification losses are defined as:

$$\begin{aligned} \begin{aligned} \mathcal {L}_{\textit{task1}}&= \textrm{CE}(\textbf{Y}, \textrm{label}), \\ \mathcal {L}_{\textit{task2}}&= \lambda _{1} \sum _{m \ne n} \textrm{CE}(\textbf{Y}_{mn}, \textrm{label}), \end{aligned} \end{aligned}$$

(13)

where $\mathcal {L}_{\textit{task2}}$ serves as an interaction-level auxiliary classification loss.

The overall task loss is:

$$\begin{aligned} \mathcal {L}_{\textit{task}} = \mathcal {L}_{\textit{task1}} + \mathcal {L}_{\textit{task2}}. \end{aligned}$$

(14)

Finally, the total training objective is formulated as:

$$\begin{aligned} \mathcal {L} = \mathcal {L}_{\textit{task}} + \lambda _{2}\mathcal {L}_{CLT}(t,v) + \lambda _{3}\mathcal {L}_{CLT}(t,a) + \lambda _{4}\mathcal {L}_{CLT}(v,a), \end{aligned}$$

(15)

where $\lambda _{1}$–$\lambda _{4}$ are hyperparameters controlling the contribution of each loss term.

Experiments

In this section, we provide a detailed description of the experimental setup for our work, including datasets, baselines, evaluation metric, and implementation details.

Dataset

We evaluate FMCL on CMU-MOSI [41], CMU-MOSEI [42] and MELD datasets. The experiments are conducted under the word-aligned setting. Table 2 details the split composition of the three datasets.

CMU-MOSI: This dataset includes 2199 brief monologue video clips. Acoustic features are sampled at 12.5 Hz, and visual features are sampled at 15 Hz.

CMU-MOSEI: This dataset has 22,856 movie review video samples from YouTube, about ten times larger than CMU-MOSI. Acoustic features in CMU-MOSEI are sampled at 20 Hz, while visual features are sampled at 15 Hz.

MELD: This dataset is a multi-party dataset derived from the TV series Friends. It consists of 13,708 utterances and 1433 dialogues. Each utterance is annotated with one of seven emotion categories: anger, disgust, fear, joy, neutral, sadness, and surprise.

Evaluation metric

Each sample in CMU-MOSI and CMU-MOSEI received a emotion score ranging from − 3 to 3, covering highly negative, negative, weakly negative, neutral, weakly positive, positive, and highly positive. The MER performance is evaluated using the following metrics:

• Acc-7: The 7-class accuracy;

• Acc-2: The binary accuracy with zero exclusion;

• MAE: The mean absolute error;

• F1 score: Provides a comprehensive performance evaluation of the model.

Implementation details

In the CMU-MOSI and CMU-MOSEI datasets, we used GloVe [43] to extract unimodal language features, resulting in 300-dimensional word features. To ensure fair comparison in the aligned setting, we also used a BERT-base-uncased pre-trained model [44] to get 768-dimensional hidden states as word features. For the visual modality, each video frame was encoded with Facet [45] to detect 35 facial action units. The acoustic modality was processed using COVAREP [46] to extract 74-dimensional features. Hyperparameters $\lambda _1$, $\lambda _2$, $\lambda _3$, and $\lambda _4$ were set to 0.5, 0.05, 0.03, and 0.05, respectively. For the MELD dataset, hyperparameters $\lambda _1$, $\lambda _2$, $\lambda _3$, and $\lambda _4$ were set to 0.5, 0.1, 0.1, and 0.05, respectively. All experiments were conducted using PyTorch on two RTX A4000 GPUs, each with 16GB of memory. The training batch size was set to 16.

Baseline

We compared FMCL with current state-of-the-art methods using the same dataset settings, including EF-LSTM, LF-LSTM, TFN [17], LMF [16], MFM [18], RAVEN [47], MCTN [48], MulT [27], MISA [49], DMD [19], Self-MI [50], CMHFM [51], TCAN [52], DLF [53], Coupled Mamba [36], DER-GCN [54].

Results and analysis

Quantitative analysis

Table 4 Results on the MELD dataset

Fig. 4

Performance analysis across 5 runs with different random seeds on the MOSI and MOSEI datasets. (a) Binary classification accuracy comparison; (b) F1 score comparison; (c) trend of 5 runs on the MOSI dataset; (d) trend of 5 runs on the MOSEI dataset

We compared our method with existing state-of-the-art approaches. Table 3 presents the corresponding results on the CMU-MOSI and CMU-MOSEI datasets. Compared to previous methods, our approach shows significant improvements across all four metrics, particularly in Acc-7 and MAE, highlighting substantial progress in fine-grained emotion classification. The performance on the CMU-MOSI dataset is particularly notable. Our significant improvements over previous works can be attributed to several key enhancements:

• Intra-modal feature extraction: By utilizing the Mamba model to further extract intra-modal features, we enhanced the representation capability of unimodal features.

• Multimodal interaction fusion mechanism: Introducing a mechanism for multimodal interaction and fusion improved the collaborative effects between different modalities. Interaction-level auxiliary classification loss: Incorporating interaction-level auxiliary classification loss during training further optimized the model’s performance.

These improvements enabled our method to perform exceptionally well across all metrics. Compared to the DMD model, which adopts a decoupling strategy, our method showed enhancements across all metrics. Particularly on the CMU-MOSI dataset, Acc-2 and F1 score improved by over 1%, strongly demonstrating the effectiveness and superiority of our proposed method.

In addition, experiments were conducted on the MELD dataset (see Table 4). This dataset exhibits an imbalanced distribution of emotion categories, with neutral and joy being overrepresented, while fear and disgust are underrepresented. Such class imbalance may adversely affect the model’s classification performance, particularly for low-frequency categories.

The experimental results demonstrate that the proposed method achieves promising performance in terms of overall classification accuracy (ACC) and weighted F1-score (W-F1), surpassing the compared methods. However, classification performance varies across different emotion categories. For instance, some baseline methods achieve better results for surprise, fear, and anger, whereas other methods perform better for disgust and sadness. This indicates that although the proposed method achieves superior overall performance, there remains room for improvement in classifying certain emotions. This phenomenon may be attributed to the dataset’s class imbalance, which causes the model to focus more on high-frequency categories during training while being less effective in distinguishing low-frequency emotions. Additionally, different emotions exhibit varying feature representations across speech, text, and visual modalities, and the current multimodal fusion strategy may not fully capture key information for certain categories. To address these limitations, future work will explore class-weighted loss functions or data augmentation strategies to enhance the classification of minority emotion classes. Furthermore, improving multimodal fusion techniques could further enhance the model’s performance across different emotion categories.

Fig. 5

T-SNE visualization of binary classification on CMU-MOSI and CMU-MOSEI datasets

Fig. 6

T-SNE visualization of 7-class classification on CMU-MOSI and CMU-MOSEI datasets

Notably, significant improvements were achieved in the seven-class classification task on the MOSI and MOSEI datasets. However, the absolute accuracy remains lower than that observed in the binary classification task. This phenomenon primarily stems from the decision-making mechanism of the binary classification: binary classification results are obtained by applying a threshold to the seven-class predictions (where values greater than 0 are considered positive and values less than 0 are considered negative). This hierarchical design leads to several key characteristics:

• Decision boundary effect: In the seven-class classification task, samples close to the decision boundary (e.g., + 1/− 1) are prone to misclassification. However, in the binary classification task, such misclassifications can often be automatically corrected—if the predicted sign is correct, the classification remains accurate.

• Difference in classification difficulty: The seven-class classification task requires the model to accurately distinguish between seven discrete sentiment intensity levels (e.g., differentiating between + 3 and + 2). In contrast, the binary classification task only involves determining the sentiment polarity. This inherently reduces the classification difficulty, resulting in a more constrained error space for binary classification compared to seven-class classification.

• Influence of subjectivity in annotation: Sentiment annotation inherently involves a high degree of subjectivity, as different annotators may assign varying sentiment scores to the same video sample. This annotation bias is particularly pronounced in the fine-grained seven-class classification task, making it more challenging for the model to learn clear category boundaries. For example, the distinction between + 2 and + 3 may depend entirely on the annotator’s personal interpretation rather than objective sentiment cues from the video itself. This subjectivity increases the difficulty of the seven-class classification task, thereby complicating fine-grained sentiment intensity modeling.

Furthermore, an overall classification accuracy (ACC) of 67.95% was achieved on the MELD dataset, significantly higher than the 48.10% on CMU-MOSI and 53.98% on CMU-MOSEI. This significant accuracy gap highlights the inherent differences between datasets in multimodal sentiment analysis tasks, demonstrating the critical impact of dataset characteristics on model performance. Additionally, this result further verifies the adaptability and robustness of the proposed method across different task scenarios. In future work, a hierarchical classification strategy will be introduced, where coarse-grained sentiment polarity (positive, neutral, and negative) is first distinguished, followed by fine-grained classification within each polarity group.

Randomness testing

The performance of multimodal sentiment recognition models can fluctuate due to random initialization. To assess the stability of our model, we conducted independent experiments on the MOSI and MOSEI datasets using five different random seeds. The experimental results demonstrate that our FMCL model exhibits good stability under different random initializations.

On the MOSI dataset, the binary classification accuracy (Acc-2) across five runs ranged from 86.43% to 86.89%, with a mean of 86.65% and a standard deviation of 0.19%. The F1 scores ranged from 86.37% to 86.83%, with a mean of 86.57% and a standard deviation of 0.18%. On the MOSEI dataset, the binary classification accuracy ranged from 85.62% to 85.95%, with a mean of 85.80% and a standard deviation of 0.14%. The F1 scores ranged from 85.50% to 85.85%, with a mean of 85.69% and a standard deviation of 0.14%.

As shown in Fig. 4, the model’s performance on both datasets shows high consistency: figures (a) and (b) display the overall performance comparison between the MOSI and MOSEI datasets, with error bars indicating standard error; figures (c) and (d) respectively show the detailed trends of the five runs on the two datasets, where solid lines connecting circular markers represent binary classification accuracy, and dashed lines connecting square markers represent F1 scores.

This stability can be attributed to several factors: (1) the selective state space mechanism of the Mamba architecture has deterministic computational properties, reducing sensitivity to initial conditions; (2) the structured multimodal fusion method provides stable learning signals; (3) contrastive learning regularization helps the model converge to similar optimization regions. These results demonstrate the reliability and reproducibility of our method under different random initialization conditions.

Visualization

Table 5 Quantitative cluster quality evaluation of t-SNE on MOSI and MOSEI

To better demonstrate the effectiveness of the proposed FMCL model, we visualized the classification results for binary and seven-class tasks on two datasets using t-SNE [55], as shown in Figs. 5 and 6. It is worth noting that while the shapes of the two plots are similar, the colors differ because the binary classification results are derived from the seven-class results. Specifically, samples with a score greater than 0 in the seven-class results are labeled as positive sentiment, whereas samples with a score less than 0 are labeled as negative sentiment.

Table 6 Performance comparison of different ablation modules on CMU-MOSI and CMU-MOSEI

Fig. 7

Comparison of average training time per epoch before and after introducing contrastive learning (CL)

Through t-SNE visualization, the superiority of the FMCL algorithm in the binary classification task is intuitively demonstrated. In this task, samples from each category are tightly distributed and easily distinguishable. However, in the seven-class task, the results are relatively poor, with overlapping clusters and difficulty in effectively separating samples from different labels. To scientifically compare the differences in the seven-class visualization, two common clustering quality metrics were computed:

• Calinski–Harabasz Score (CHS): This metric reflects the compactness within clusters and the separation between clusters. A higher value indicates better clustering performance. A higher CHS suggests stronger separation between different categories and a more compact distribution of samples within the same category.

• Davies–Bouldin Score (DBS): This metric measures the similarity between clusters, where a lower value indicates better clustering performance. A lower DBS suggests better distinguishability between clusters and higher similarity within each cluster.

Table 7 Performance comparison of different ablated modalities on CMU-MOSI and CMU-MOSEI

Fig. 8

Robustness comparison of different models under partial text modality missing scenarios on the MOSI dataset. The percentage drop (shown in red for FMCL) is calculated relative to the model’s performance at 30% missing rate. A smaller drop indicates better robustness

Table 8 The best result in each column is highlighted in bolditalic and bold

Fig. 9

Performance comparison of Mamba and Transformer based on the DMD model on the CMU-MOSI dataset

Fig. 10

Comparison of parameter numbers and inference speed between Mamba and Transformer

Fig. 11

Comparison of GPU memory usage and training speed between Mamba and Transformer

Fig. 12

Performance of different numbers of Mamba blocks on CMU-MOSI

As shown in Table 5, the FMCL method shows significant improvement over the original features in these metrics. Specifically, for both the MOSI and MOSEI datasets, FMCL achieves significantly lower DBS values and significantly higher CHS values compared to the original features, indicating that our method has notably improved clustering performance, especially in the binary classification task. However, in the seven-class task, despite improvements over the original features, issues such as category overlap and insufficient separation persist. This suggests that further enhancements in feature representation or the introduction of additional strategies may be necessary to improve clustering performance in multi-class classification tasks.

Ablation study

Ablation of FMCL

To evaluate the effectiveness of our proposed model, we removed each key module and critical loss function. Specifically, we removed the Intra-Mamba module, Tri-Modal Fusion Mamba (TFM), and Contrastive Learning (CL) module from the complete FMCL model; for critical loss functions, we removed the contrastive learning loss ($\mathcal {L}_{CLT}$) and the interaction-level auxiliary classification loss ($\mathcal {L}_{\textrm{task2}}$), which is applied to the cross-modal interaction representations $\textbf{H}_{m \rightarrow n}$ rather than the original unimodal features.

The ablation study results in Table 6 demonstrate that each component of the FMCL model contributes to its overall performance. Specifically, the TFM module and the Intra-Mamba module play key roles in enhancing the effectiveness of multimodal feature fusion and classification. In the CMU-MOSI dataset, removing the TFM module results in the most significant performance degradation, whereas in the CMU-MOSEI dataset, removing the Intra-Mamba module has the greatest impact on the Acc-7 metric.

In addition, removing the interaction-level auxiliary classification loss $\mathcal {L}_{\textrm{task2}}$ consistently leads to performance drops on both datasets, indicating that explicitly supervising intermediate cross-modal interaction representations is beneficial for learning task-discriminative features.

Table 9 Performance of different numbers of Mamba blocks on CMU-MOSI

Although the removal of the contrastive learning module and its corresponding loss also leads to performance degradation, their impact is relatively smaller compared to the core fusion modules. Notably, the best performance is achieved when the interaction-level auxiliary supervision and contrastive learning are jointly employed, suggesting that these two components are complementary in improving the robustness and generalization of multimodal representations.

In this study, we introduced tri-modal contrastive learning (CL) based on the Mamba architecture, which added a certain level of complexity to the model. Notably, since contrastive learning did not introduce additional training parameters, only minor changes in memory usage were observed during the experiments. To further evaluate its impact on training efficiency, we compared the training time before and after incorporating contrastive learning.

Fig. 13

Comparison of removing the SSM module

The results in Fig. 7 show that after incorporating contrastive learning, the average training time per epoch increased from 28.13 to 29.21 s, representing an increase of approximately 1.08 s (about 3.84%). Although this increase is relatively small, it reflects the additional computational burden associated with calculating similarities or differences between data from different modalities in contrastive learning.

Fig. 14

Visualization of different configurations without the SSM module on CMU-MOSI

Modality ablation and robustness analysis

To investigate the contribution of each modality and the model’s robustness under realistic conditions, we conduct ablation studies with both complete modality removal and partial modality missing scenarios.

Complete modality removal. In Table 7, we individually remove each modality to explore the performance of the dual-modality FMCL. Compared to the tri-modal FMCL, the dual-modality FMCL consistently performs worse, indicating that each modality provides indispensable information. Notably, when the text modality is completely removed, the model’s performance degrades significantly, while removing the visual or auditory modalities results in only a minor performance drop. This confirms that the text modality is the most crucial for sentiment analysis, likely because it contains richer and more explicit semantic information compared to the other two modalities.

Table 10 Performance of different configurations without the SSM module on CMU-MOSI

Partial text modality missing (robustness analysis). While the complete removal experiments show the presence of each modality is essential, real-world applications often face scenarios where modalities are partially available or corrupted. Since text is the dominant modality, its degradation most severely impacts performance. To assess model robustness under such realistic conditions, we simulate partial text modality missing on the MOSI dataset and compare against several strong baselines.

The results are shown in Fig. 8 and Table 8. As the text missing ratio increases from 30% to 90%, all models’ performance declines. However, our FMCL model demonstrates superior robustness, particularly in maintaining a high F1 score. For instance, with 90% of the text missing, our model achieves an F1 score of 59.9%, significantly higher than other baselines (50.9% or lower). In terms of absolute change, the F1 score of our model decreases by only 18.8 points (from 78.7% at 30% missing to 59.9% at 90% missing), which is the smallest among all compared methods. This indicates that our fusion strategy and contrastive learning objective enable the model to better utilize the remaining modality information and maintain a good precision-recall balance even when the key modality is severely compromised.

Analysis and practical implications. The complete ablation highlights the necessity of all modalities, especially text. The partial missing experiments further demonstrate the robustness of our model’s architecture. This robustness stems from: (1) the selective state space mechanism of Mamba, which dynamically focuses on relevant features from available data; (2) the multimodal contrastive learning, which strengthens cross-modal alignment and representation quality; and (3) our structured fusion strategy, which does not overly rely on any single modality being complete.

These findings are crucial for practical deployment. In real applications, textual information can be partially lost or corrupted due to reasons such as automatic speech recognition (ASR) errors, data transmission issues, or incomplete user input. Our model’s ability to maintain reasonable performance under such conditions makes it more reliable and applicable for real-world multimodal sentiment analysis systems.

Fig. 15

Case of model

Fig. 16

Case of positive sentiment

Fig. 17

Case of negative sentiment

Ablation of Mamba

To further verify the effectiveness of the Mamba module, we compared our proposed FMCL with the state-of-the-art DMD model. Specifically, we used DMD as the baseline, where DMD employed Transformer. For comparison, we replaced the Transformer in DMD with Mamba. For single-modality Transformers, we used Intra-Mamba as a replacement, and for Multimodal Transformers, we used Fusion Mamba as a replacement. The results are shown in Fig. 9.

Figure 6 shows the performance metrics (Acc-2, Acc-7, F1, and MAE) over training epochs, comparing Mamba and Transformer implementations. The visualized results indicate consistent improvements when using Mamba, thus affirming its effectiveness in enhancing MER.

As shown in Fig. 10, we compared the number of parameters and inference speed of Mamba and Transformer. The results indicate that Mamba has significantly fewer parameters than Transformer while providing higher performance. Mamba’s parameter count is 858,750, whereas Transformer’s parameter count is 2,560,500, a reduction of 66.5%. This demonstrates a significant advantage in model size for Mamba. In terms of inference speed, Mamba also outperforms Transformer. Over 19 epochs, Mamba’s average inference time is 4.05 s, compared to Transformer’s average inference time of 5.98 s, an improvement of 32.2%. This highlights Mamba’s significantly enhanced efficiency in practical applications.

As shown in Fig. 11, we compared the GPU memory usage and training speed of Mamba and Transformer during training. The experimental results indicate that Mamba achieves faster training while consuming less GPU memory. Specifically, Mamba’s GPU memory usage is 4.06 GB, whereas Transformer requires 5.86 GB, representing a reduction of approximately 30.7%, highlighting Mamba’s significant advantage in memory efficiency. Moreover, Mamba also outperforms Transformer in terms of training efficiency. After 19 training epochs, the average training time for Mamba is 28.13 s, compared to 38.12 s for Transformer, reflecting a 26.21% improvement. These findings demonstrate the advantages of Mamba in efficient training and resource utilization, providing valuable insights for optimizing DL models.

Ablation of Mamba layers

To investigate the impact of Mamba blocks, we explored how different numbers of Mamba blocks behave in terms of model performance.

Figure 12 and Table 9 illustrate the performance variations of different numbers of Mamba blocks across various metrics on the validation set. As the number of Mamba blocks increases, Acc-2 and F1 scores show a significant improvement. However, after reaching 3 Mamba blocks, the performance gain starts to diminish, and in some cases, the improvement even levels off. Acc-7 follows a similar trend, and MAE achieves the best results with 6 Mamba blocks.

Based on the above results, we identified that the optimal number of Mamba blocks lies between 3 and 6. With this range, we can achieve the best overall performance. However, employing 6 Mamba blocks, while providing a slight advantage over 3 Mamba blocks, requires significantly more computational memory. Therefore, for practical applications, balancing computational efficiency and performance, selecting an appropriate number of Mamba blocks is crucial.

Ablation of SSM

To investigate the impact of the SSM module, we conducted ablation experiments, as shown in Fig. 13, by removing the SSM module and comparing the performance before and after its removal. This analysis aims to explore the role of the SSM module in MER. Previous work [60] has examined whether the SSM module significantly contributes to visual tasks; therefore, we extend this investigation to MER.

By comparing the data in Table 10 and Fig. 14, it is evident that the inclusion of the SSM module significantly improves the model’s performance across various metrics. Specifically, Acc-2 and F1-score both increase by approximately 1%, Acc-7 increases by 1.45%, and MAE decreases by about 0.0169. This indicates that the SSM module plays a crucial role in enhancing model accuracy and reducing error.

Further analysis reveals that the SSM module effectively captures the similarity between multimodal data, improving feature extraction accuracy, and thus yielding better performance in emotion recognition tasks. Therefore, retaining the SSM module is essential to ensure the model’s efficiency and accuracy in practical applications.

Case study

This section validates the FMCL at the example level. We selected two examples from CMU-MOSI for positive and negative emotions. As shown in Fig. 15, we perform emotion prediction in different parts of the whole framework. Specifically, the original logit is obtained by simply fusing the original features first, then the Intra logit is obtained by fusing the three modalities from the Intra-Mamba posterior, and finally the FMCL logit is obtained from the full model.We compare with the real labels in the examples.

As shown in Figs. 16 and 17, these two emotion recognition examples demonstrate the performance of the model in recognising positive and negative emotions. The text in Fig. 16 is a sentence with positive sentiment, indicating that the person holds a positive evaluation of the performance; the text in Fig. 17 is a sentence with negative sentiment, indicating that the person holds a negative evaluation of something. By comparing the Origin logit, Intra logit and FMCL logit with Ground truth, it can be seen that the FMCL logit is closer to Ground truth, which shows that it is more accurate in emotion recognition, and more importantly, it shows the effectiveness of the various modules of FMCL.

Conclusion

This paper proposes a novel emotion recognition algorithm (FMCL) based on multimodal fusion Mamba and contrastive learning. To effectively extract different features, we use Intra-Mamba to enhance unimodal representations. Simultaneously, to achieve comprehensive information fusion, we extend the Mamba block to accommodate dual inputs, creating a new module called the Two-Fusion Mamba block and design a tri-modal, tri-branch fusion architecture on this basis. Additionally, we introduce contrastive learning and interaction-level auxiliary classification to further enhance the performance. Experimental results on three public datasets demonstrate the superiority of our method. In future work, we will explore more feasible schemes for Mamba.

Discussion on real-world applicability

In practical applications, the FMCL model, by integrating information from video, audio, and text modalities, demonstrates outstanding potential for applicability across multiple fields. For instance, in health monitoring, the model can capture patients’ facial expressions from video data, extract emotional fluctuations from audio signals, and combine text data (e.g., medical records or self-reported symptoms) to provide robust support for comprehensive health evaluations. In intelligent customer service, the FMCL model can identify customers’ identities and emotional states through video analysis, quickly interpret customer needs via speech recognition, and integrate text-based information to deliver more precise solutions, thereby enhancing the overall customer experience. In educational settings, video data can be used to capture students’ classroom performance, audio data can record teacher-student interactions, and text data can facilitate the analysis of assignments and feedback, providing a basis for personalized teaching and real-time instructional adjustments. Furthermore, on social media platforms, by comprehensively analyzing multimodal information from posts related to emerging events, the FMCL model can integrate various data types to promptly reflect public sentiment toward an incident, offering valuable insights for public opinion monitoring and the formulation of response strategies. Overall, the FMCL model’s strengths in multimodal information fusion not only enhance task efficiency across diverse domains but also lay a solid theoretical and practical foundation for future cross-domain applications and system integration.