Multi modality fusion transformer with spatio-temporal feature aggregation module for psychiatric disorder diagnosis.

Journal: Computerized medical imaging and graphics : the official journal of the Computerized Medical Imaging Society

Volume: 114

Issue: 

Year of Publication: 2024

Affiliated Institutions:  College of Biomedical Engineering & Instrument Science, Zhejiang University, Hangzhou , China. Beijing Huilongguan Hospital, Peking University Huilongguan Clinical Medical School, Beijing , China. College of Sciences, Northeastern University, Shenyang , China. JD Health International Inc., Beijing , China. College of Biomedical Engineering & Instrument Science, Zhejiang University, Hangzhou , China. Electronic address: osfengyu@zju.edu.cn. College of Sciences, Northeastern University, Shenyang , China. Electronic address: wangqimath@mail.neu.edu.cn. Beijing Huilongguan Hospital, Peking University Huilongguan Clinical Medical School, Beijing , China. Electronic address: zhiren@.com.

Abstract summary 

Bipolar disorder (BD) is characterized by recurrent episodes of depression and mild mania. In this paper, to address the common issue of insufficient accuracy in existing methods and meet the requirements of clinical diagnosis, we propose a framework called Spatio-temporal Feature Fusion Transformer (STF2Former). It improves on our previous work - MFFormer by introducing a Spatio-temporal Feature Aggregation Module (STFAM) to learn the temporal and spatial features of rs-fMRI data. It promotes intra-modality attention and information fusion across different modalities. Specifically, this method decouples the temporal and spatial dimensions and designs two feature extraction modules for extracting temporal and spatial information separately. Extensive experiments demonstrate the effectiveness of our proposed STFAM in extracting features from rs-fMRI, and prove that our STF2Former can significantly outperform MFFormer and achieve much better results among other state-of-the-art methods.

Authors & Co-authors:  Wang Fan Shi An Cao Ge Yu Wang Han Tan Tan Wang

Study Outcome 

Source Link: Visit source

Statistics
Citations : 
Authors :  12
Identifiers
Doi : 10.1016/j.compmedimag.2024.102368
SSN : 1879-0771
Study Population
Male,Female
Mesh Terms
Humans
Other Terms
Bipolar disorder;Magnetic resonance imaging;Medical diagnosis;Multimodal deep learning;Spatio-temporal feature aggregation module
Study Design
Study Approach
Country of Study
Publication Country
United States