Black-Box Problem Localization in Parallel File Systems

Parallel file systems target large, high-performance storage systems. Since these storage systemsare comprised of a significant number of components (i.e., hundreds of file servers, thousands ofdisks, etc.), they are expected to (and in practice do) frequently exhibit “problems”, from degradedperformance to outright failure of one or more components. The sheer number of components,and thus, potential problems, makes manual diagnosis of these problems difficult. Of particularconcern are system-wide performance degradations, which may arise from a single misbehavingcomponent, and thus, pose a challenge for problem localization. Even failure of a redundant componentwith a less-significant performance impact is worrisome as it may, in absence of explicitchecks, go unnoticed for some time and increase risk of system unavailability.As a solution, this thesis defines a novel problem-diagnosis approach, capitalizing upon theparallel-file-system design criterion of balanced performance, that peer-compares the performanceof system components to localize problems within storage systems running unmodified, “off-theshelf”parallel file systems. Performed in support of this thesis is a set of laboratory experimentsthat demonstrate proof-of-concept of the peer-comparison approach by injecting four realisticproblems into 12-server, test-bench PVFS and Lustre clusters. This thesis is further validatedby taking the diagnosis approach and adapting it to to work on a very-large, production GPFS storagesystem consisting of 128 file servers, 32 storage controllers, 1152 disk arrays, and 11,520 totaldisks. Presented in a 15-month case study is the problems observed through analysis of 624GB ofinstrumentation data, in which a variety of performance-related storage-system problems are localizedand diagnosed, in a matter of hours, as compared to days or longer with manual approaches