Decentralized DNN Architectures

bg-new
Author/s

Ioannis Pitas (AUTH)

About the resource/s

This lecture overviews decentralized and distributed DNN architectures. Big data analysis can be greatly facilitated if decentralized/distributed DNN architectures are employed that interact with each other for DNN training and/or inference using the human Teacher-Student education paradigm. A novel Learning-by-Education Node Community (LENC) framework is presented that facilitates communication and knowledge exchange among diverse Deep Neural Networks (DNN) agents, undertaking the role of a student or teacher DNN by offering or absorbing knowledge respectively. The framework enables efficient and effective knowledge transfer among participating DNN agents while enhancing their learning capabilities and fostering their collaboration among diverse networks. The proposed framework addresses the challenges of handling diverse training data distributions and the limitations of individual DNN agent learning abilities. The LENC framework ensures the exploitation of the best available teacher knowledge upon learning a new task and protects the DNN agents from catastrophic forgetting. The experiments demonstrate the LENC framework functionalities on multiple teacher-student learning techniques and their integration with lifelong learning. Our experiments manifest the LEMA framework’s ability to maximize the accuracy of all participating DNN agents in classification tasks by leveraging the collaborative knowledge of the framework. The LENC framework also addresses the problem of task-agnostic lifelong learning as DNN agents have no information on task boundaries.

Other Sources