Towards Distributed Message Queues using Distributed Key/Value Stores

Abstract: In today’s world, distributed message queues are used in many systems and play different roles (e.g. content delivery, notification system and message delivery tools). It is important for the queue services to be able to deliver messages at large scales with a variety of message sizes with high concurrency. An example of a commercial state of the art distributed message queue is Amazon Simple Queuing Service (SQS). SQS is a distributed message delivery fabric that is highly scalable. It can queue unlimited number of short messages (maximum size: 256 KB) and deliver them to multiple users in parallel. In order to be able to provide such high throughput at large scales, SQS omits some of features that are provided by traditional queues. SQS does not guarantee the order of the messages, nor does it guarantee the exactly once delivery. This report addresses these limitations through the design and implementation of ZDMQ, a distributed message queue using distributed key-value store. ZDMQ consist of collection of ZHT server that can be used to store messages up to 1 MB message size. ZDMQ provides replication of messages for high reliability. In the preliminary testing we performed evaluation and compared ZDMQ to the commonly used commercial distributed queues measuring throughput and latency. We found ZDMQ to outperform SQS, HDMQ, Windows Azure Service bus, and IronMQ by up to 1.86-351x times in throughput and 3.6-177x times in latency. Keywords-distributed message queues, exactly-once delivery, distributed key-value store, zero-hop distributed hash table. I