Forecasting weekly crude oil using Twitter sentiment of US foreign policy and oil companies data

Abstract The drop in crude oil price during late 2014 has had a significant impact on all nations. While some countries have reaped the benefits of low oil prices, others have suffered greatly. As a result, it is no surprise that many academics have attempted to develop reliable models to forecast crude oil price. In the age of information and social media, the role of Twitter and Facebook has become increasingly more relevant in understanding our environment. Many academics have exploited this wealth of data to extract features including sentiment and word frequency to build reliable forecasting models for financial instruments such as stocks. These methodologies, however, remain unexplored for the prediction of crude oil prices. The purpose of this investigation to develop a novel model that uses sentiment of United States foreign policy and oil companies’ to forecast the direction of weekly WTI crude oil prices. The investigation is divided into three parts: 1) a methodology of collecting tweets relevant to US foreign policy and oil companies’; 2) a statistical analysis of the novel features using Granger Causality Test; 3) the development and evaluation of three machine learning classifiers including Naïve Bayes, ANNs, and SVM to predict the direction of weekly WTI crude oil. The findings of the statistical analysis showed strong correlation between the novel inputs and WTI crude oil price. The results of the statistical tests were then used in the development of the predictive model. SVM was found to provide best forecasting performance. Furthermore, using these novel features, the predictive accuracy exceeded that of existing models mentioned in literature