Deep Learning for Dynamic Multiple Object Recognition on the Shareish Solidarity Platform

Abstract

Shareish is an interactive map-based web platform developed with the intention of encouraging people to show more solidarity. An important design requirement of the platform is the efficiency of user content creation. Recent research in artificial intelligence has made it possible to use deep learning to provide various auto-completion mechanisms, such as object classification, to help users create more structured content. Indeed, the need for users to encode metadata can be a barrier to the use of the platform. In this master thesis, we aimed at building an efficient object detection model that would recognize multiple objects from an uploaded image. To this end, we compared two different paradigms: closed-set and open-vocabulary object detection. The first step was to build a dynamic dataset that would cover as many categories as possible for training and evaluating a first closed-set object detection algorithm: Faster R-CNN. Even though the results were quite encouraging, this category of model lacks flexibility with new additional classes. To overcome this issue, we evaluated two open-vocabulary object detection models: Grounding DINO and Segment Anything 3, which have the advantage of being zero-shot. The performance on the giving box dataset showed very promising results, indicating that vision-language models are well suited for this project.