Predicting Intermediate Storage Performance for Workflow Applications

Abstract. System provisioning, resource allocation, and system con-figuration decisions for workflow I/O-intensive applications are complex even for expert users. One factor making the problem space more com-plex is that users face decisions at multiple levels: allocating resources to individual sub-systems (e.g., the application layer, the storage layer) and configuring each of these optimally (e.g., replication level, chunk size, caching policies in case of storage) which have a large impact on overall application performance. This paper presents our progress on addressing the problem of support-ing these provisioning, allocation and configuration decisions for work-flow applications. To enable selecting a good choice in a reasonable time, we propose an approach that accelerates the exploration of the config-uration space based on a low-cost performance predictor that estimates total execution time of a workflow application in a given setup. The eval-uation shows that we are on a good track to meet our objectives: our performance predictor can scale to model a workflow application run on an entire cluster while offering an over 200x speedup compared to run-ning the actual application, and can achieve, in the limited number of scenarios we study, a prediction accuracy that enables identifying the best storage system configuration.