IBIS: Interposed Big-data I/O Scheduler

Existing big-data systems (e.g., Hadoop/MapReduce) do not expose management of shared storage I/O resources. As such, application’s performance may degrade in unpre-dictable ways under I/O contention, even with fair sharing of computing resources. This paper proposes IBIS, a new Interposed Big-data I/O Scheduler, to provide performance differentiation for competing applications ’ I/Os in a shared MapReduce-type big-data system. IBIS is implemented in Hadoop by interposing HDFS I/Os and use an SFQ-based proportional-sharing algorithm. Experiments show that the IBIS provides strong performance isolation for one applica-tion against another highly I/O-intensive application. IBIS also enforces good proportional sharing of the global band-width among competing parallel applications, by coordinat-ing distributed IBIS schedulers to deal with the uneven dis-tribution of local services in big-data systems