HLS4ML for Computer Vision: A Study of Potential and Limitations

Abstract

The goal of this thesis is to explore the capabilities of the HLS4ML ( [11]) library for the Euresys company. This library has been created by the CERN to implement real time low latency neural network on FPGA. My work in this thesis was to explore the capabilities of this library for Euresys. The idea of this research comes from one of their client that, with their help, implemented a real time video inference neural network to stabilize plasma in a tokamak reactor. This work implemented a small neural network that predicted the resonance amplitude of a plasma and its phase in order to stabilize it. This neural network was implemented on a KCU035 FPGA from Xilinx on a frame grabber from Euresys. This frame grabber used the custom logic framework developed by the company. This framework allows user to implement their custom on-board image processing directly on the FPGA. I was asked by Euresys to explore the HLS4ML library on their custom logic board with the objective of finding the capabilities and limitations of this tools inside of their hardware. The goal was to explore what HLS4ML could enable for their client. The main idea was that HLS4ML allows someone to program an FPGA using python which could be helpful as it bypass the need to learn RTL, synthesis and place and route tools. The objective of this thesis was to: • Find the limitation of the library with the custom logic framework. • Explore the possibilities to implement a python code directly to an FPGA thanks to the HLS4ML library. • Explore different network topologies and their FPGA implementation • Find some useful application of implementing neural network on FPGA instead of CPU or GPU. The scope of this thesis was to implement computer vision algorithm. Euresys being a company that specialise in empowering computer vision with the help of acquisition card and software.