Master thesis : Content-aware retargeting of broadcast videos

Abstract

Video retargeting, or the challenge of transforming a video from one aspect ratio to another, has become a source of great interest in recent years. While the de-facto standard for filming productions has been 16:9 for a long time, the growth of social media and the broadening of screen sizes demand for an automatic conversion procedure. With this thesis, we provide an overview of the current practices for this field both in the literature and in the industry. We discuss why one dimensional cropping should be preferred over other hybrid techniques in the context of the broadcast industry.

Resulting from this study, we introduce our own modular framework composed of two subsequent computational blocs. On one hand, the first module comprises a state-of-the-art video saliency detection model which locates and quantifies relevant information. As part of our contributions, we build our own saliency dataset called EVS-Sal and fine-tune the deep network to specialize its detections for soccer content. On the other hand, the second module is responsible for the selection of cropped salient information while ensuring temporal consistency. For this purpose, we explore both global and local optimizations respectively with the dynamic programming paradigm and with a “select and filter” approach.

Finally, we show that our methods outperform current one dimensional retargeting algorithms on a variety of general videos. Additionally, we extend this analysis with the creation of our own soccer retargeting dataset called EVS-Ret. With the latter, we demonstrate that our framework brings results near inter-human agreement and that the semantics of soccer are correctly captured by the re-trained saliency model.