Load balancing of heterogeneous workloads in memcached clusters

Web services, large and small, use in-memory caches like memcached to lower database loads and quickly re-spond to user requests. These cache clusters are typi-cally provisioned to support peak load, both in terms of request processing capabilities and cache storage size. This kind of worst-case provisioning can be very expen-sive (e.g., Facebook reportedly uses more than 10,000 servers for its cache cluster) and does not take advantage of the dynamic resource allocation and virtual machine provisioning capabilities found in modern public and pri-vate clouds. Further, there can be great diversity in both the workloads running on a cache cluster and the types of nodes that compose the cluster, making manual man-agement difficult. This paper identifies the challenges in designing large-scale self-managing caches. Rather than requiring all cache clients to know the key to server mapping, we propose an automated load balancer that can perform line-rate request redirection in a far more dynamic manner. We describe how stream analytic tech-niques can be used to efficiently detect key hotspots. A controller then guides the load balancer’s key mapping and replication level to prevent overload, and automati-cally starts additional servers when needed.