Master thesis : 3D synthetic data generation in the waste sorting field

Abstract

Deep Learning models need quality labeled data and a huge amount of samples to train. While open datasets exist, it is not usually the case for specific projects. Recent advancements showed that state-of-the-art synthetic data generation can be used to tackle the large data issue and generate infinite synthetic data samples (e.g. GANs, Diffusion models, 3D data augmentation, etc.). In this master thesis, we explore 3D data augmentation techniques for 2D synthetic data generation for cutlery classification and waste sorting. Given a set of 3D objects, the use of a 3D graphics engine (e.g. Unity) allows us to generate various versions of a scene by changing the color, texture, scale, camera angles, lighting conditions, etc. While creating these random scenes, all the labels and segmentation masks could be computed on the fly, allowing to generate an unlimited set of scene-label examples that will be used to train the model. In this work, we study the theory and implement our own protocol based on domain randomization. We design a board in the Unity software to generate synthetic data. From these synthetic datasets, we provide experiments to compare the different training cases (i.e. only synthetic, only real, both real and synthetic) to investigate the potential of the 3D synthetic data generation when only few data is available. These experiments suggest that there is a real potential behind the 3D data augmentation techniques. At the end of this work, we highlight the main improvements that could be made on our synthetic generator.