APPLICATION-AWARE OPTIMIZATIONS FOR BIG DATA ACCESS

Many High-Performance Computing (HPC) applications spend a significant portion of their execution time in accessing data from les and they are becoming increasingly data-intensive. For them, I/O performance is a significant bottleneck leading to wastage of CPU cycles and the corresponding wasted energy consumption. Various optimization techniques exist to improve data access performance. However, the existing general-purpose optimization techniques are not able to satisfy diverse applications' demands. On the other hand, the application-specific optimization pro- cess is usually a difficult task due to the complexity involved in understanding the parallel I/O system and the applications' I/O behaviors. To address these challenges, this thesis proposes an application-aware data access optimization framework and claims that it is feasible and useful to utilize applications' characteristics to improve the performance and efficiency of the parallel I/O system. Under this framework, an optimization may consist of several basic but challenging steps, including capturing the application's characteristics, identifying the causality of I/O performance degra- dation, and delivering optimization solutions. To make these steps easier, we design and implement the IOSIG toolkit as an essential system support for the default par- allel I/O system. The toolkit is able to pro le the applications' I/O behaviors and then generate comprehensive characteristics through trace analysis. With the help of IOSIG, we design several optimization techniques on data layout optimization, data reorganization, and I/O scheduling. The proposed framework has significant poten- tial to boost application-aware I/O optimization. The results prove that the proposed optimization techniques can significantly improve the data access performance.Ph.D. in Computer Science, July 201