TaskMaster: A Scalable, Reliable Queuing Infrastructure for Building Distributed Systems

TaskMaster is a system for managing priority-ordered queues that is designed to scale to 1 billion tasks across 100 thousand queues per node. A reliable queuing sys-tem, such as TaskMaster, provides a mechanism for dis-tributing units of inherently serial work (tasks) to work-ers. Priorities are lexicographically ordered strings that give users more power than FIFO or fixed-range inte-ger priorities when defining queue order. TaskMaster provides the ability to atomically transition a task be-tween queues, facilitating the decomposition of process-ing into a sequence of tasks. Individual tasks are exclu-sively leased to workers for the duration of processing to minimize system-wide work duplication. TaskMas-ter helps system designers detect bottlenecks and direct system optimization by providing queue statistics such as enqueue and dequeue rates, as well as queue lengths. TaskMaster allows application designers to focus on the individual stages of data processing instead of task dis-tribution.