Engineering efficient event-based distributed systems

Event-based design inherently decouples software components and enables distributed systems to scale to a large number of components. Event-based distributed systems have to be engineered to disseminate, detect and respond to simple and complex events in a scalable and efficient manner. Engineering complex event-based distributed applications often goes through combining different middleware systems, which are treated as black boxes by the programming language in which the application logic is implemented, decreasing performance. This decrease in performance is more pronounced in the case of applications that change their subscriptions at a high frequency, e.g., algorithmic trading. This dissertation uses the EventJava framework, consisting of the EventJava programming language and two middleware systems—the Beretta content-based publish/subscribe system and the GenTrie event correlation system—to demonstrate that the efficiency of event-based distributed applications can be increased by aligning middleware systems with specialized programming abstractions. EventJava is an extension of Java with advanced support for event-based distributed programming, including event correlation, unicast and multicast of events. The EventJava compiler uses definitions of complex events to generate an optimized complex event detection component, called GenTrie, at each event consumer. GenTrie consists of an event flow graph, that is designed to discard unwanted events early and to effectively store events that partially match correlation patterns. The EventJava compiler leverages strong event typing and represents all subscriptions in a normalized form as a combination of value intervals and set inclusions without compromising on expressiveness. This, in turn, is leveraged by Beretta, to (1) split subscriptions into predicates based on individual event types and attributes and to efficiently regroup these in interval trees and hash maps yielding excellent subsumption properties, and (2) improve the efficiency of matching events to subscriptions for routing events from publishers to subscribers. Normalization also enables the EventJava compiler to systematically introduce variables into predicates to efficiently perform subscription updates, which are vital to emerging event-based distributed applications in high frequency algorithmic trading, mobile social networking, traffic management, etc. In our empirical evaluation using real world and statistical benchmarks, we demonstrate that the use of the EventJava framework can significantly increase the performance of event-based distributed applications. We have used a real world algorithmic trading benchmark for performance evaluation, by identifying all system and workload parameters and varying them to demonstrate the scalability of our techniques