Published In ARGUS: Rete + DBMS = Efficient Persistent Profile Matching on Large-Volume Data Streams

Abstract. Efficient processing of complex streaming data presents multiple chal-lenges, especially when combined with intelligent detection of hidden anoma-lies in real time. We label such systems Stream Anomaly Monitoring Systems (SAMS), and describe the CMU/Dynamix ARGUS system as a new kind of SAMS to detect rare but high value patterns combining streaming and historical data. Such patterns may correspond to hidden precursors of terrorist activity, or early indicators of the onset of a dangerous disease, such as a SARS outbreak. Our method starts from an extension of the RETE algorithm for matching streaming data against multiple complex persistent queries, and proceeds beyond to transi-tivity inferences, conditional intermediate result materialization, and other such techniques to obtain both accuracy and efficiency, as demonstrated by the evalu-ation results outperforming classical techniques such as a modern DMBS.