A feature set for streams and an application to high-frequency financial tick data

We propose a set of features to study the effects of data streams on complex systems. This feature set is called the the signature representation of a stream. It has its origin in pure mathematics and relies on a relationship between non-commutative polynomials and paths. This representation had already signifcant impact on algebraic topology, control theory, numerics for PDEs, stochastic analysis and the theory of rough paths; more recently first steps have been taken to apply such methods to the study of big data streams. We show that the signature representation can provide an efficient summary of a stream and its effects. We then show that it can be combined with standard tools from machine learning. After introducing the signature for streams and some theoretical background, we apply this approach to a challenging real-world example: high-frequency financial data streams. In this context, the streams are tick-by-tick market data of a stock traded at the New York stock exchange NYSE and the effect of the stream is the profit and loss of complex investment strategies (i.e. a nonlinear functional of the stream). Our numerical results (applied to Thomson--Reuters tick data of several full trading days for IBM stocks) show that the signature of the price stream efficiently captures the necessary information to learn the return of an investment strategy. However, we emphasize that the underlying ideas are not limited to financial data streams and have the potential to be applied to many other areas in data mining where the non-commutative nature of streams is of importance, like text mining, bioinformatics or click history