Daily stock market forecast from textual web data

Data mining can be described as "making better use of data". Every human being is increasingly faced with unmanageable amounts of data, hence, data mining or knowledge discovery apparently affects all of us. It is therefore recognized as one of the key research areas. Ideally, we would like to develop techniques for "making better use of any kind of data for any purpose". However, we argue that this goal is too demanding yet. It may sometimes be more promising to develop techniques applicable to specific data and with a specific goal in mind. In this paper, we describe such an application driven data mining system. Our aim is to predict stock markets using information contained in articles published on the Web. Mostly textual articles appearing in the leading and influential financial newspapers are taken as input. From those articles the daily closing values of major stock market indices in Asia, Europe and America are predicted. Textual statements contain not only the effect (e.g. the stocks plummet) but also why it happened (e.g. because of weakness in the dollar and consequently a weakening of the treasury bonds). Exploiting textual information in addition to numeric time series data increases the quality of the input. Hence improved predictions are expected. The forecasts are available real-time via www.cs.ust.hk/similar to beat/Predict daily at 7:45 am Hong Kong time. Hence all predictions are ready before Tokyo, Hong Kong and Singapore, the major Asian markets, start trading. The system's accuracy for this tremendously difficult but also extremely challenging application is highly promising