Dynamic Attention-Driven Fusion of Multi-Scale CNN and Transformer Networks for Facial Expression Recognition.
Main Article Content
Abstract
Reliable facial expression recognition (FER) is essential for information and communication technology applications such as human-computer interaction, affective computing, driver monitoring, and smart surveillance. Conventional CNN-based FER methods capture local facial patterns effectively but often struggle with long-range facial-region dependencies, pose variation, occlusion, illumination changes, and subtle expressions. This study proposes FusionX-Net: DAF-RealFER, which integrates a multi-scale CNN, a transformer encoder, and a dynamic attention mechanism to combine local detail, global context, and expression-sensitive region weighting. The model was evaluated on FER-2013 and AffectNet using accuracy, precision, recall, F1-score, Cohen's Kappa, AUC-ROC, inference time, statistical significance tests, and ablation analysis. It achieved 92.45% accuracy on FER-2013 and 85.85% on AffectNet; the complete model obtained 92.1% precision, 92.4% recall, 91.9% F1-score, and 0.96 AUC-ROC in the reported ablation setting. The findings show that attention-guided CNN-transformer fusion improves robustness and interpretability compared with the evaluated baseline variants, while remaining suitable for selected real-time ICT applications.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
References
P. Ekman and W. V. Friesen, “Constants across cultures in the face and emotion,” Journal of Personality and Social Psychology, vol. 17, no. 2, pp. 124–129, 1971, doi: 10.1037/h0030377.
I. J. Goodfellow et al., “Challenges in representation learning: A report on three machine learning contests,” Neural Networks, vol. 64, pp. 59–63, Apr. 2015, doi: 10.1016/j.neunet.2014.09.005.
P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews, “The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression,” in Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition Workshops (CVPRW), 2010, pp. 94–101, doi: 10.1109/CVPRW.2010.5543262.
E. Barsoum, C. Zhang, C. C. Ferrer, and Z. Zhang, “Training deep networks for facial expression recognition with crowd-sourced label distribution,” in Proc. 18th ACM Int. Conf. Multimodal Interaction, 2016, pp. 279–283, doi: 10.1145/2993148.2993165.
A. Mollahosseini, B. Hasani, and M. H. Mahoor, “AffectNet: A database for facial expression, valence, and arousal computing in the wild,” IEEE Transactions on Affective Computing, vol. 10, no. 1, pp. 18–31, Jan.–Mar. 2019, doi: 10.1109/TAFFC.2017.2740923.
S. Li, W. Deng, and J. Du, “Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2852–2861, doi: 10.1109/CVPR.2017.277.
S. Li and W. Deng, “Deep facial expression recognition: A survey,” IEEE Transactions on Affective Computing, vol. 13, no. 3, pp. 1195–1215, Jul.–Sep. 2022, doi: 10.1109/TAFFC.2020.2981446.
F. Z. Canal, T. R. Müller, J. C. Matias, G. G. Scotton, A. R. de Sa Junior, E. Pozzebon, and A. C. Sobieranski, “A survey on facial emotion recognition techniques: A effective literature review,” Information Sciences, vol. 582, pp. 593–617, Jan. 2022, doi: 10.1016/j.ins.2021.10.005.
K. Wang, X. Peng, J. Yang, S. Lu, and Y. Qiao, “Suppressing uncertainties for large-scale facial expression recognition,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2020, pp. 6897–6906, doi: 10.1109/CVPR42600.2020.00693.
A. H. Farzaneh and X. Qi, “Facial expression recognition in the wild via deep attentive center loss,” in Proc. IEEE Winter Conf. Applications of Computer Vision (WACV), 2021, pp. 2401–2410, doi: 10.1109/WACV48630.2021.00245.
Z. Zhao, Q. Liu, and S. Wang, “Learning deep global multi-scale and local attention features for facial expression recognition in the wild,” IEEE Transactions on Image Processing, vol. 30, pp. 6544–6556, 2021, doi: 10.1109/TIP.2021.3093397.
F. Ma, B. Sun, and S. Li, “Facial expression recognition with visual transformers and attentional selective fusion,” IEEE Transactions on Affective Computing, vol. 14, no. 2, pp. 1236–1248, Apr.–Jun. 2023, doi: 10.1109/TAFFC.2021.3122146.
C. Liu, K. Hirota, and Y. Dai, “Patch attention convolutional vision transformer for facial expression recognition with occlusion,” Information Sciences, vol. 619, pp. 781–794, Jan. 2023, doi: 10.1016/j.ins.2022.11.068.
Y. Tian, Z. Wang, D. Chen, and H. Yao, “TriCAFFNet: A tri-cross-attention transformer with a multi-feature fusion network for facial expression recognition,” Sensors, vol. 24, no. 16, Art. no. 5391, 2024, doi: 10.3390/s24165391.
H. Li, N. Wang, Y. Yu, X. Yang, and X. Gao, “LBAN-IL: A novel method of high discriminative representation for facial expression recognition,” Neurocomputing, vol. 432, pp. 159–169, Apr. 2021, doi: 10.1016/j.neucom.2020.12.076.
X. Xia, L. Yang, X. Wei, H. Sahli, and D. Jiang, “A multi-scale multi-attention network for dynamic facial expression recognition,” Multimedia Systems, vol. 28, no. 2, pp. 479–493, 2022.
B. Yang, J. Wu, K. Ikeda, G. Hattori, M. Sugano, Y. Iwasawa, and Y. Matsuo, “Face-mask-aware facial expression recognition based on face parsing and vision transformer,” Pattern Recognition Letters, vol. 164, pp. 173–182, Dec. 2022, doi: 10.1016/j.patrec.2022.11.004.
Y. Tian, Y. Zhang, D. Chen, and H. Yao, “Facial expression recognition based on vision transformer and hybrid local attention,” Applied Sciences, vol. 14, no. 15, Art. no. 6471, 2024.
Y. Zhang, C. Wang, and W. Deng, “Relative uncertainty learning for facial expression recognition,” in Proc. Advances in Neural Information Processing Systems (NeurIPS), vol. 34, 2021, pp. 17616–17627.
J. She, Y. Hu, H. Shi, J. Wang, Q. Shen, and T. Mei, “Dive into ambiguity: Latent distribution mining and pairwise uncertainty estimation for facial expression recognition,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2021, pp. 6248–6257, doi: 10.1109/CVPR46437.2021.00618.
F. Zhang, G. Chen, H. Wang, and C. Zhang, “CF-DAN: Facial-expression recognition based on cross-fusion dual-attention network,” Computational Visual Media, vol. 10, pp. 593–608, 2024, doi: 10.1007/s41095-023-0369-x.
H. Li, N. Wang, X. Yang, and X. Gao, “Unconstrained facial expression recognition with no-reference de-elements learning,” IEEE Transactions on Affective Computing, vol. 15, no. 1, pp. 173–185, Jan.–Mar. 2024, doi: 10.1109/TAFFC.2023.3263886.
H. Shin, B. Lee, B. Ku, and H. Ko, “Noisy label facial expression recognition via face-specific label distribution learning,” Image and Vision Computing, vol. 143, Art. no. 104901, Mar. 2024, doi: 10.1016/j.imavis.2024.104901.
H. Li, N. Wang, X. Yang, X. Wang, and X. Gao, “Towards semi-supervised deep facial expression recognition with an adaptive confidence margin,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2022, pp. 4166–4175, doi: 10.1109/CVPR52688.2022.00413.
H. Li, N. Wang, X. Yang, and X. Gao, “CRS-CONT: A well-trained general encoder for facial expression analysis,” IEEE Transactions on Image Processing, vol. 31, pp. 4637–4650, 2022, doi: 10.1109/TIP.2022.3186536.
H. Li, X. Yuan, C. Xu, R. Zhang, X. Liu, and L. Liu, “Complexity aware center loss for facial expression recognition,” The Visual Computer, vol. 40, pp. 8045–8054, 2024, doi: 10.1007/s00371-023-03221-1.
J. Jiang and W. Deng, “Disentangling identity and pose for facial expression recognition,” IEEE Transactions on Affective Computing, vol. 13, no. 4, pp. 1868–1878, Oct.–Dec. 2022.
R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-CAM: Visual explanations from deep networks via gradient-based localization,” in Proc. IEEE Int. Conf. Computer Vision (ICCV), 2017, pp. 618–626, doi: 10.1109/ICCV.2017.74