An Extended Two-Phase Method for Accessing Sections of Out-of-Core Arrays

thakurmcsanlgov choudharcatsyredu A number of applications on parallel computers deal with very large data sets that cannot t in main memory In such applications data must be stored in les on disks and fetched into memory during program execution Parallel programs with large outofcore arrays stored in les must readwrite smaller sections of the arrays fromto les In this paper we describe a method for accessing sections of outofcore arrays eciently Our method the extended twophase method uses collective IO Processors cooperate to combine several IO requests into fewer larger granularity requests reorder requests so that the le is accessed in proper sequence and eliminate simultaneous IO requests for the same data In addition the IO workload is divided among processors dynamically depending on the access requests We present performance results obtained from two real outofcore parallel applicationsmatrix multiplication and a Laplaces equation solverand several synthetic access patterns all on the Intel Touchstone Delta These results indicate that the extended twophase method signicantly outperformed a direct noncollective method for accessing outofcore array section