Multimodal contrastive learning for brain–machine fusion: From brain-in-the-loop modeling to brain-out-of-the-loop application
Abstract
Harnessing the complementarity between the brain and artificial neural networks has shown great promise for the development of novel brain–machine fusion systems that may rival the robustness and flexibility of the human visual system. Nonetheless, due to the requirement of human involvement, brain–machine fusion models are challenging to apply in a brain-in-the-loop manner to handle task demands that involve long duration, high intensity and complex operating environments. In this paper, we propose a brain–machine fusion approach to achieve the brain-in-the-loop modeling and brain-out-of-the-loop application. The similarities and differences between the image representations of brain responses and image features computed by deep convolutional neural network (DCNN) are firstly analyzed using a rhesus monkey dataset, and the results lay the foundation for the feasibility of brain–machine fusion. A brain–machine fusion model is then developed and a multimodal supervised contrastive learning method is proposed to jointly learn the image representations for brain responses and DCNNs. The fusion model can be applied in a brain-out-of-the-loop manner, effectively addressing the challenges encountered by human-involved approaches. Extensive experiments on a self-built human vehicle detection dataset demonstrate the effectiveness of the proposed method in improving the generalization ability of the downstream image classifiers, both in cross-modal learning or multimodal fusion settings.
Kaitai Guo
Assistant Professor
My research interests include broad-spectrum substance identification, microwave and infrared imaging, and system simulation and evaluation.
Yang Zheng
Assistant Professor
My research interests include human behaviour analysis for intelligent diagnosis of developmental coordination disorder, aritifical intelligence, and computer vision.
Minghao Dong
Professor of Biomedical Engineering
My research interests include intelligent human-computer interaction, brain science and neuroimaging.
Jimin Liang
Professor of Electronic Engineering
My research interests include artificial intelligence and computer vision.