Towards a distributed implementation of a data extraction system for monitoring the processing of video clips

Abstract

The objective of this work was to review the data handling of an application monitoring the processing of video clips and to remove any barrier preventing the concurrent execution of multiple instances of the service, while maintaining or improving the overall performance.

The main obstacle to concurrent execution was that the caching infrastructure didn’t provide synchronisation between the cache and the database. We showed that the database could perform queries in memory like the in-memory cache at the server. The database executed most of the queries in a more efficient and predictable way than the cache, thanks to its query optimiser.

Based on those results, we decided to remove the cache and perform queries directly on the database. This removal simplified and shortened the code of the application. However, there was no mechanism of translation of the query contained in the URI to the database. We thus used OData, a tool providing a query language for RESTful APIs, that can translate automatically query parameters from the URI to SQL queries. Yet, OData came with some performance flaws. To restore performances, we used indexes to speed up queries execution and we restricted query parameters combinations.

This work resulted in a solution allowing the concurrent execution of multiple instances of the service. Moreover, queries are in most cases faster and thus the service can now support more clients.

So, this work showed that the cache was not better than the database when executing the queries and could be removed.