Locality Aware MPI Communication on a Commodity Opto-Electronic Hybrid Network

Future supercomputers with millions of processors would pose significant challenges in their interconnection networks due to difficulty in design constraints such as space, cable length, cost, power consumption, etc. In-stead of huge switches or bisection bandwidth restricted topologies such as a torus, we propose a network which utilizes both fully-connected lower-bandwidth electronic packet switching (EPS) network and low-power optical cir-cuit switching (OCS) network. Optical circuits, connected sparingly to only a limited set of nodes to conserve power and cost, are used in a supplemental fashion as “shortcut” routes only when a node communicates substantially across EPS switches, while short latency communication is han-dled by EPS only. Our MPI inter-node communication al-gorithm accommodates for such a network by appropriate scheduling of nodes according to application communica-tion patterns, in particular utilizing relatively high EPS lo-cal switch bandwidth to forward messages to nodes with optical connections for shortcutting in order to maximize overall throughput. Simulation studies confirm that our pro-posal effectively avoids contentions in the network in high-bandwidth applications with nominal additions of optical circuitry to existing machines. 1