Scalable monitoring of a highly dynamic metric set

Metric collection and analysis is an important aspect of operational management of many systems. Adequate monitoring can help reduce the mean time to recovery in failure scenarios, as well as guide both long term and short term system capacity planning. Many existing monitoring software focuses on metrics collection on a per-host basis. Over the past years, an increasing number of systems is being built on top of cloud infrastructure. The notion of a host is drastically changing from the original concept of a physical machine towards an ephemeral compute unit. In addition, typical workloads are no longer pinned to a fixed set of these hosts, but dynamically move around systems based on properties such as available resources. Per-user monitoring in this type of system, such as VMware's Cloud Foundry, proves to be difficult using existing monitoring software. In this work, I will present an outline of existing monitoring software and its typical deficiencies when applied to systems. Next, I will present a new data model that allows monitoring, storage and analytics for metrics in highly dynamic systems.