Feedback

Faculté des Sciences appliquées
Faculté des Sciences appliquées
MASTER THESIS
VIEW 114 | DOWNLOAD 212

Bistable Recurrent Cells and Belief Filtering for Q-learning in Partially Observable Markov Decision Processes

Download
Lambrechts, Gaspard ULiège
Promotor(s) : Ernst, Damien ULiège
Date of defense : 24-Jun-2021/25-Jun-2021 • Permalink : http://hdl.handle.net/2268.2/11474
Details
Title : Bistable Recurrent Cells and Belief Filtering for Q-learning in Partially Observable Markov Decision Processes
Translated title : [fr] Cellules récurrentes bistables et filtrage de la distribution sur les états pour le Q-learning dans les processus de décisions markoviens partiellement observables
Author : Lambrechts, Gaspard ULiège
Date of defense  : 24-Jun-2021/25-Jun-2021
Advisor(s) : Ernst, Damien ULiège
Committee's member(s) : Louppe, Gilles ULiège
Drion, Guillaume ULiège
Bolland, Adrien ULiège
Language : English
Number of pages : 74
Keywords : [en] Reinforcement Learning
[en] Belief Filtering
[en] POMDP
[en] Deep Recurrent Q-Network
[en] DRQN
[en] Online Fitted Q-Iteration
[en] OFQI
[en] RNN
[en] Bistable Recurrent Cell
[en] BRC
[en] Q-Learning
[en] Markov Decision Process
[en] MDP
[en] Bistability
[en] Target Network
[en] Partially Observable Markov Decision Process
[en] Recurrent Neural Network
[en] Mutual Information
[en] RL
Discipline(s) : Engineering, computing & technology > Computer science
Target public : Researchers
Professionals of domain
Student
Institution(s) : Université de Liège, Liège, Belgique
Degree: Master : ingénieur civil en science des données, à finalité spécialisée
Faculty: Master thesis of the Faculté des Sciences appliquées

Abstract

[en] In this master's thesis, reinforcement learning (RL) methods are used to learn (near-)optimal policies to act in several Markov decision processes (MDPs) and partially observable Markov decision processes (POMDPs). More precisely, Q-learning and recurrent Q-learning techniques are used. Some of the considered POMDPs require a high-memorisation ability in order to achieve optimal decision making. In POMDPs, RL techniques usually rely on approximating functions that take as input sequences of observations with variable length. Recurrent neural networks (RNNs) are thus a clever choice of such approximators. This work is based on the recently introduced bistable recurrent cells, which have been empirically shown to provide a significantly better long term memory than standard cells, such as the long short-term memory (LSTM) and the gated recurrent unit (GRU). These cells are named the bistable recurrent cell (BRC) and the recurrently neuromodulated BRC (nBRC). First, by importing these cells for the first time in the RL setting, it is empirically shown that they also provide a significant advantage in memory-demanding POMDPs, in comparison to LSTM and GRU. Second, the ability of the RNN to represent a belief distribution over the states of the POMDP is studied. It is achieved by evaluating the mutual information between the hidden states of the RNN and the belief filtered on the successive observations. This analysis is thus strongly anchored in the theory of information and the theory of optimal control for POMDPs. Third, as a complement to this research project, a new target update is proposed for Q-learning algorithms with target networks, for both reactive and recurrent policies. This new update speeds up learning, especially in environments with sparse rewards.


File(s)

Document(s)

File
Access thesis.pdf
Description: Master's Thesis
Size: 3.97 MB
Format: Adobe PDF
File
Access summary.pdf
Description: Master's Thesis Summary
Size: 167.72 kB
Format: Adobe PDF

Author

  • Lambrechts, Gaspard ULiège Université de Liège > Master ingé. civ. sc. don. à . fin.

Promotor(s)

Committee's member(s)

  • Louppe, Gilles ULiège Université de Liège - ULiège > Dép. d'électric., électron. et informat. (Inst.Montefiore) > Big Data
    ORBi View his publications on ORBi
  • Drion, Guillaume ULiège Université de Liège - ULiège > Dép. d'électric., électron. et informat. (Inst.Montefiore) > Systèmes et modélisation
    ORBi View his publications on ORBi
  • Bolland, Adrien ULiège Université de Liège - ULiège > Dép. d'électric., électron. et informat. (Inst.Montefiore) > Smart grids
    ORBi View his publications on ORBi
  • Total number of views 114
  • Total number of downloads 212










All documents available on MatheO are protected by copyright and subject to the usual rules for fair use.
The University of Liège does not guarantee the scientific quality of these students' works or the accuracy of all the information they contain.