Use of Flink Hybrid sources to initialise transformations

Abstract

This master's thesis explores using Apache Flink's HybridSource technology to enhance the data transformation processes within the Digazu framework, a data processing platform developed by EuraNova. The thesis addresses the complexities of transitioning from batch to stream processing, traditionally managed through custom orchestration solutions. By leveraging HybridSource, which facilitates seamless data ingestion from multiple heterogeneous sources, the study aims to reduce orchestration code complexity, optimize resource usage, and improve latency performance.

The research involves a comprehensive literature review on streaming frameworks, a technical analysis of the HybridSource implementation in Apache Flink, and empirical validation through proof-of-concept experiments. By practically implementing HybridSource within Digazu's infrastructure, the study evaluates its feasibility and effectiveness in streamlining data transformation processes. This investigation demonstrates significant reductions in orchestration code complexity and optimized resource usage and highlights improvements in latency performance.