Towards a distributed implementation of a data extraction system for monitoring the processing of video clips
Vasbinder, Thomas
Promotor(s) : Wolper, Pierre
Date of defense : 27-Jun-2016/28-Jun-2016 • Permalink : http://hdl.handle.net/2268.2/1306
Details
Title : | Towards a distributed implementation of a data extraction system for monitoring the processing of video clips |
Author : | Vasbinder, Thomas |
Date of defense : | 27-Jun-2016/28-Jun-2016 |
Advisor(s) : | Wolper, Pierre |
Committee's member(s) : | Boigelot, Bernard
Donnet, Benoît Latour, Louis |
Language : | English |
Discipline(s) : | Engineering, computing & technology > Computer science |
Institution(s) : | Université de Liège, Liège, Belgique |
Degree: | Master en ingénieur civil en informatique, à finalité approfondie |
Faculty: | Master thesis of the Faculté des Sciences appliquées |
Abstract
[en] The objective of this work was to review the data handling of an application monitoring the processing of video clips and to remove any barrier preventing the concurrent execution of multiple instances of the service, while maintaining or improving the overall performance.
The main obstacle to concurrent execution was that the caching infrastructure didn’t provide synchronisation between the cache and the database. We showed that the database could perform queries in memory like the in-memory cache at the server. The database executed most of the queries in a more efficient and predictable way than the cache, thanks to its query optimiser.
Based on those results, we decided to remove the cache and perform queries directly on the database. This removal simplified and shortened the code of the application. However, there was no mechanism of translation of the query contained in the URI to the database. We thus used OData, a tool providing a query language for RESTful APIs, that can translate automatically query parameters from the URI to SQL queries. Yet, OData came with some performance flaws. To restore performances, we used indexes to speed up queries execution and we restricted query parameters combinations.
This work resulted in a solution allowing the concurrent execution of multiple instances of the service. Moreover, queries are in most cases faster and thus the service can now support more clients.
So, this work showed that the cache was not better than the database when executing the queries and could be removed.
File(s)
Document(s)
Annexe(s)
Description: Program parsing measured times (client, middleware, controller) from logs
Size: 2.2 kB
Format: Unknown
Description: Modified version of the MultiMonitoring application
Size: 1.71 MB
Format: Unknown
Description: Initial version of the MultiMonitoring application
Size: 1.72 MB
Format: Unknown
Description: Program allowing to send queries to the server a given number of times.
Size: 1.35 kB
Format: Unknown
Description: Gatling script used to measure the number of clients (OData version)
Size: 3.51 kB
Format: Unknown
Description: Gatling script used to measure the number of clients (cache version)
Size: 3.29 kB
Format: Unknown
Description: SQL script creating 132 indexes on the Jobs table
Size: 14.62 kB
Format: Unknown
Cite this master thesis
The University of Liège does not guarantee the scientific quality of these students' works or the accuracy of all the information they contain.