Master Thesis : Online Distillation with Continual Learning for Cyclic Domain Shifts

Abstract

The technique of online distillation has become increasingly popular in adapting real- time deep neural networks using a slow and accurate teacher model. However, one of the most significant challenges encountered with online distillation is catastrophic forgetting, which happens when the student model is updated with new domain data and loses the previously learned knowledge. The main contribution in this thesis is to apply continual learning techniques to mitigate the problem fo catastrophic forgetting in online distillation. Indeed, continual learning has shown to be useful in a more general setting, where the model tend to forget knowledge from previous tasks when learning the current one. The study aims to assess the efficacy of various state-of-the-art continual learning methods in reducing catastrophic forgetting when applied to online distillation, particularly in cyclic domain shifts. The experimental results presented in this study show improved accuracy and robustness in the context of online distillation when leveraging continual learning methods to reduce catastrophic forgetting, with potential applications in fields such as video surveillance or autonomous driving. As such, this work represents a significant contribution to the fields of online distillation and continual learning, providing new insights and avenues for future research.