Predicting the houseprice index using sentiment analysis and machine learning

Abstract

The 2008 financial crisis showed that economists seem to be far from a complete understanding of the real estate market fluctuations. There is often a strong gap between the variations of fundamental economic and demographic factors and the actual housing price changes.. An alternative approach to the study real estate market fluctuations, called Behavioral Real Estate, emerged from the evidence of unexplained volatility and market inefficiency. It deviates from the traditional way of studying the real estate market by explaining changes in house prices with traditional factors like inflation, interest rates and construction costs. This study focuses on forecasting the housing market and seeks to provide new evidence on the role of sentiment in driving house prices, which, to date, is a fairly underestimated approach. The difficulty in measuring the sentiment regarding the real estate market has made it really challenging to explain volatility in the housing market, despite clear evidence of psychological factors in explaining price fluctuations. Throughout the thesis, we followed the hypothesis that public opinion on social media has an impact on the behavior of the real estate market participants and subsequently affects future price trends. To test our hypothesis, we have trained a model with economic variables to predict the London House Price Index (HPI) using recurrent neural networks. Then, we have built a tweet-based sentiment index for the London housing market and incorporated it as a feature in our initial model. The model trained with economic and sentiment data performs better (MAE = 0.8315 and RMSE = 1.0358) than the one with economic data alone (MAE = 0.9012 and RMSE = 1.1701). Moreover, the permutation feature importance showed that the sentiment index is a significant feature in our predictive model. These results confirm our initial beliefs that the incorporation of a sentiment indicator should improve the overall performance of the model. This implies that real estate forecasters, policymakers and professional investors should consider incorporating this free data into their market predictions or when deciding for future investments.