Faculté des Sciences appliquées
Faculté des Sciences appliquées
VIEW 288 | DOWNLOAD 3471

Master's Thesis : NetBERT: A Pre-trained Language Representation Model for Computer Networking

Louis, Antoine ULiège
Promotor(s) : Louppe, Gilles ULiège
Date of defense : 25-Jun-2020/26-Jun-2020 • Permalink :
Title : Master's Thesis : NetBERT: A Pre-trained Language Representation Model for Computer Networking
Translated title : [fr] NetBERT : Un Modèle de Représentation Linguistique pour le Domaine des Réseaux Informatiques.
Author : Louis, Antoine ULiège
Date of defense  : 25-Jun-2020/26-Jun-2020
Advisor(s) : Louppe, Gilles ULiège
Committee's member(s) : Mathy, Laurent ULiège
Geurts, Pierre ULiège
De Pra, Hugues 
Language : English
Number of pages : 94
Discipline(s) : Engineering, computing & technology > Computer science
Target public : Researchers
Professionals of domain
Institution(s) : Université de Liège, Liège, Belgique
Degree: Master : ingénieur civil en science des données, à finalité spécialisée
Faculty: Master thesis of the Faculté des Sciences appliquées


[en] Obtaining accurate information about products in a fast and efficient way is becoming increasingly important at Cisco as the related documentation rapidly grows. Thanks to recent progress in natural language processing (NLP), extracting valuable information from general domain documents has gained in popularity, and deep learning has boosted the development of effective text mining systems. However, directly applying the advancements in NLP to domain-specific documentation might yield unsatisfactory results due to a word distribution shift from general domain language to domain-specific language. Hence, this work aims to determine if a large language model pre-trained on domain-specific (computer networking) text corpora improves performance over the same model pre-trained exclusively on general domain text, when evaluated on in-domain text mining tasks.

To this end, we introduce NetBERT (Bidirectional Encoder Representations from Transform-ers for Computer Networking), a domain-specific language representation model based on BERT (Devlin et al., 2018) and pre-trained on large-scale computer networking corpora. Through several extrinsic and intrinsic evaluations, we compare the performance of our novel model against the general-domain BERT. We demonstrate clear improvements over BERT on the following two representative text mining tasks: networking text classification (0.9% F1 improvement) and networking information retrieval (12.3% improvement on a custom retrieval score). Additional experiments on word similarity and word analogy tend to show that NetBERT capture more meaningful semantic properties and relations between networking concepts than BERT does. We conclude that pre-training BERT on computer networking corpora helps it understand more accurately domain-related text.



Access Antoine_Louis_Thesis.pdf
Size: 14.05 MB
Format: Adobe PDF


Size: 17.74 MB
Format: Unknown
Size: 1.24 MB
Format: Unknown
Access Antoine_Louis_summary.pdf
Size: 74.68 kB
Format: Adobe PDF


  • Louis, Antoine ULiège Université de Liège > Master ingé. civ. sc. don. à . fin.


Committee's member(s)

  • Mathy, Laurent ULiège Université de Liège - ULiège > Dép. d'électric., électron. et informat. (Inst.Montefiore) > Systèmes informatiques répartis et sécurité
    ORBi View his publications on ORBi
  • Geurts, Pierre ULiège Université de Liège - ULiège > Dép. d'électric., électron. et informat. (Inst.Montefiore) > Algorith. des syst. en interaction avec le monde physique
    ORBi View his publications on ORBi
  • De Pra, Hugues
  • Total number of views 288
  • Total number of downloads 3471

All documents available on MatheO are protected by copyright and subject to the usual rules for fair use.
The University of Liège does not guarantee the scientific quality of these students' works or the accuracy of all the information they contain.