Catching up with the data: Research issues in mining data streams

In many domains, data now arrives faster than we are able to mine it. To avoid wasting this data, we must switch from the traditional \one-shot " data mining approach to systems that are able to mine continuous, high-volume, open-ended data streams as they arrive. In this paper we identify some desiderata for such systems, report on our recent work realizing them, and outline a number of directions for future research. 1 The Problem Many (or most) organizations today produce an electronic record of essentially every trans-action they are involved in. When the organization is large, this results in millions of records being produced every day. For example, in a single day WalMart records 20 million sales transactions, Google handles 70 million searches, and AT&T produces 275 million call records. Scientic data collection (e.g., by earth sensing satellites or astronomical observa-tories) routinely produces gigabytes of data per day. Data rates of this level have signicant consequences for data mining. For one, a few months ' worth of data can easily add up to billions of records, and the entire history of transactions or observations can be in the hun