Large Language Models: Building General Coding Assistants

Abstract

In the realm of software development, the frequent release of new Application Programming Interface (API) versions presents a significant challenge for engineers and developers. Traditionally, adapting to these changes requires a comprehensive update of the entire application, resulting in considerable time and resource investments. This situation highlights the need to support developers in managing the numerous tedious tasks they encounter daily.

This thesis addresses these challenges by leveraging Large Language Models (LLMs) for code-related tasks and introduces a framework for deploying advanced general coding assistants that achieve state-of-the-art performance.

The approach involves selecting and deploying a model based on several meaningful criteria, choosing appropriate benchmarks and datasets for fine-tuning, and developing a framework capable of fine-tuning on a single GPU. We also deploy our own benchmark, building upon the dataset released in previous related works.

We address the limitations associated with fine-tuning under constrained computational resources. Our fine-tuned models demonstrate a systematic improvement in performance for the specific downstream tasks they are adapted to. Improving their precision up to 206.25\%.

We also provide critical insights into both the evaluation metrics for LLMs and the limits of current benchmarks.