Uploaded by AUTH

Machine learning and deep learning models are the main engines in many multimodal AI applications, which are characterized by the fusion of multiple modalities of data streams. In this lecture, we highlight the trust and robustness challenges of machine learning that arises from data fusion. To do so, we present several case studies demonstrating how multimodal applications exacerbate existing challenges of trustworthy and robust machine learning. In a first case study, we investigate the impact of fusion depth on the robustness of multi-modal machine learning models, observing that model architecture could impact robustness. In a second case study, we investigate the impact of fusion modality on the robustness of multi-modal machine learning models, observing that fusion models are only as robust as their most susceptible modality. In another case study, we explore the impact of weight quantization techniques on the robustness of multimodal models, observing the need for modality-based quantization schemes. Through these case studies, we hope to share some perspectives on the unique trust and security challenges that arise in AI machine learning models in typical multimodal applications and offer insights to fortify such systems in real-world scenarios.