AtCAF: Attention-based causality-aware fusion network for multimodal sentiment analysis

Changqin Huang, Jili Chen, Qionghao Huang, Shijin Wang, Yaxin Tu, Xiaodi Huang

Research output: Contribution to journalArticlepeer-review

Abstract

Multimodal sentiment analysis (MSA) involves interpreting sentiment using various sensory data modalities. Traditional MSA models often overlook causality between modalities, resulting in spurious correlations and ineffective cross-modal attention. To address these limitations, we propose the Attention-based Causality-Aware Fusion (AtCAF) network from a causal perspective. To capture a causality-aware representation of text, we introduce the Causality-Aware Text Debiasing Module (CATDM) utilizing the front-door adjustment. Furthermore, we employ the Counterfactual Cross-modal Attention (CCoAt) module integrate causal information in modal fusion, thereby enhancing the quality of aggregation by incorporating more causality-aware cues. AtCAF achieves state-of-the-art performance across three datasets, demonstrating significant improvements in both standard and Out-Of-Distribution (OOD) settings. Specifically, AtCAF outperforms existing models with a 1.5% improvement in ACC-2 on the CMU-MOSI dataset, a 0.95% increase in ACC-7 on the CMU-MOSEI dataset under normal conditions, and a 1.47% enhancement under OOD conditions. CATDM improves category cohesion in feature space, while CCoAt accurately classifies ambiguous samples through context filtering. Overall, AtCAF offers a robust solution for social media sentiment analysis, delivering reliable insights by effectively addressing data imbalance. The code is available at https://github.com/TheShy-Dream/AtCAF.
Original languageEnglish
Article number102725
JournalInformation Fusion
DOIs
Publication statusPublished - 02 Oct 2024

Fingerprint

Dive into the research topics of 'AtCAF: Attention-based causality-aware fusion network for multimodal sentiment analysis'. Together they form a unique fingerprint.

Cite this