Master's Thesis : NetBERT: A Pre-trained Language Representation Model for Computer Networking
Promotor(s) : Louppe, Gilles
Date of defense : 25-Jun-2020/26-Jun-2020 • Permalink :
|Title :||Master's Thesis : NetBERT: A Pre-trained Language Representation Model for Computer Networking|
|Translated title :||[fr] NetBERT : Un Modèle de Représentation Linguistique pour le Domaine des Réseaux Informatiques.|
|Author :||Louis, Antoine|
|Date of defense :||25-Jun-2020/26-Jun-2020|
|Advisor(s) :||Louppe, Gilles|
|Committee's member(s) :||Mathy, Laurent
De Pra, Hugues
|Number of pages :||94|
|Discipline(s) :||Engineering, computing & technology > Computer science|
|Target public :||Researchers|
Professionals of domain
|Institution(s) :||Université de Liège, Liège, Belgique|
|Degree:||Master : ingénieur civil en science des données, à finalité spécialisée|
|Faculty:||Master thesis of the Faculté des Sciences appliquées|
[en] Obtaining accurate information about products in a fast and efficient way is becoming increasingly important at Cisco as the related documentation rapidly grows. Thanks to recent progress in natural language processing (NLP), extracting valuable information from general domain documents has gained in popularity, and deep learning has boosted the development of effective text mining systems. However, directly applying the advancements in NLP to domain-specific documentation might yield unsatisfactory results due to a word distribution shift from general domain language to domain-specific language. Hence, this work aims to determine if a large language model pre-trained on domain-specific (computer networking) text corpora improves performance over the same model pre-trained exclusively on general domain text, when evaluated on in-domain text mining tasks.
To this end, we introduce NetBERT (Bidirectional Encoder Representations from Transform-ers for Computer Networking), a domain-specific language representation model based on BERT (Devlin et al., 2018) and pre-trained on large-scale computer networking corpora. Through several extrinsic and intrinsic evaluations, we compare the performance of our novel model against the general-domain BERT. We demonstrate clear improvements over BERT on the following two representative text mining tasks: networking text classification (0.9% F1 improvement) and networking information retrieval (12.3% improvement on a custom retrieval score). Additional experiments on word similarity and word analogy tend to show that NetBERT capture more meaningful semantic properties and relations between networking concepts than BERT does. We conclude that pre-training BERT on computer networking corpora helps it understand more accurately domain-related text.
Cite this master thesis
The University of Liège does not guarantee the scientific quality of these students' works or the accuracy of all the information they contain.