Trading off space for passes in graph streaming problems

Data stream processing has recently received increasing attention as a computational paradigm for dealing with massive data sets. While major progress has been achieved for several fundamental data sketching and statistics problems, there are many problems that seem to be hard in a streaming setting, including most classical graph problems. Relevant examples are graph connectivity and shortest paths, for which linear lower bounds on the "space x passes" product are known. Some recent papers have shown that several graph problems can be solved with one or few passes, if the working memory is large enough to contain all the vertices of the graph. A natural question is whether we can reduce the space usage at the price of increasing the number of passes. Surprisingly, no algorithm with both sublinear space and passes is known for natural graph problems in classical streaming models. Motivated by technological factors of modern storage systems, some authors have recently started to investigate the computational power of less restrictive streaming models. In a FOCS'04 paper, Aggarwal et al. have shown that the use of intermediate temporary streams, combined with the ability to reorder them at each pass for free, yields enough power to solve in polylogarithmic space and passes a variety of problems, including graph connectivity. They leave however as an open question whether problems such as shortest paths can be solved efficiently in this more powerful model. In this paper, we show that the "streaming with sorting" model by Aggarwal et al. can yield interesting results even without using sorting at all: by just using intermediate temporary streams, we provide the first effective spacepasses tradeoffs for natural graph problems. In particular, for any space restriction of s bits, we show that single-source shortest paths in directed graphs with small positive integer edge weights can be solved in O((n log(3/2) n)/root 7s) passes. This is the first known streaming algorithm for shortest paths in directed graphs. For undirected connectivity, we devise an O((n log n)/s) passes algorithm. Both problems require Omega(n/s) passes under the restrictions we consider. We also show that the model where intermediate temporary streams are allowed can be strictly more powerful than classical streaming for some problems, while maintaining all of its hardness for others