Quantification of audio/video de-synchronization with reliable confidence scores

Abstract

Audio-video desynchronization remains a significant challenge in broadcast and multimedia applications, as even slight desynchronization can negatively affect viewer experience. This master's thesis focuses on the evaluation and improvement of Synchformer, a state-of-the-art classifier model for estimating temporal offsets for 'in the wild' video content. A second focus is placed on assessing uncertainty associated with predictions of the model.

To achieve finer and more reliable predictions, Synchformer was fine-tuned for temporal offset regression. These fine-tuned models offer consistent performance across the full range of temporal offsets, addressing a limitation of the original discrete-class approach. In parallel, various confidence measures were explored to quantify prediction certainty. Some measures are based on predictions from Synchformer, while others use a history of predictions to assess the consistency of the model over time. Based on these measures, classification techniques were developed to distinguish between reliable and unreliable predictions.

Experiments conducted on SyncST, a newly introduced dataset of over 1500 broadcast video clips, showed that, when considering discrete offset, it is possible to classify reliable predictions with a precision of 92% for Synchformer in classification, corresponding to an error tolerance of 200ms.

In a regression setting, where offsets are continuous and the error tolerance is 170ms to align with the humanly undetectable limits, this reliability classification achieves a precision of 75% on fine-tuned versions of Synchformer. While, when using an error tolerance of 275ms, which is an acceptable error for viewers, it is possible to achieve a precision of 90%. Considering the low performance of these models at predicting desynchronization below what is humanly perceivable or acceptable, these contributions improve the trustworthiness that could be put in such a synchronization model, making it more applicable to real-world applications where low error tolerance and interpretability are critical.