When One Pipeline Is Not Enough

Pipelines that operate on buffers often work well to mitigate the high latency inherent in interprocessor communication and in accessing data on disk. Running a single pipeline on each node works well when each pipeline stage consumes and produces data at the same rate. If a stage might consume data faster or slower than it produces data, a single pipeline becomes unwieldy. We describe how we have extended the FG programming environment to support multiple pipelines in two forms. When a node might send and receive data at different rates during interprocessor communication, we use disjoint pipelines that send and receive on each node. When a node consumes and produces data from different streams on the node, we use multiple pipelines that intersect at a particular stage. Experimental results for two out-of-core sorting algorithms---one based on columnsort and the other a distribution-based sort---demonstrate the value of multiple pipelines