DPFS: A distributed parallel file system

One of challenges brought by large-scale scientific ap-plications is how to avoid remote storage access by collec-tively using enough local storage resources to hold huge amount of data generated by the simulation while provid-ing high performance I/O. DPFS, a Distributed Parallel File System, is designed and implemented to address this problem. DPFS collects locally distributed unused stor-age resources as a supplement to the internal storage of parallel computing systems to satisfy the storage capac-ity requirement of large-scale applications. In addition, like parallel file systems, DPFS provides striping mecha-nisms that divides a file into small pieces and distributes them across multiple storage devices for parallel data ac-cess. The unique feature of DPFS is that it provides three file levels with each file level corresponding to a file strip-ing method. In addition to the traditional linear strip-ing method, DPFS also provides a novel Multidimensional striping method that can solve performance problems of lin-ear striping for many popular access patterns. Other is-sues such as load-balanceing and user interface are also addressed in DPFS.