A predictive model for Stream Processing System that dynamically calibrates the number of operator replicas

International audienceBased on direct acyclic graphs, where vertices and edges respectively correspond to operators and event data flow, Stream Processing Systems (SPSs) are used to process high amounts of data in real time. Operators are allocated in the resources of the infrastructure (e.g., VMs) which are usually replicated for performance sake. We propose in this paper a predictive SPS that dynamically determines the current number of replicas required for each operator based not only on the current resource utilization and data flow variation but also on the events that, due to operator's overloading, could not be processed yet and are, thus, kept in the operator's queue. Preliminary performance results with an application that processes Twitter stream, deployed on Google Cloud Platform (GCP), are presented