2014 IEEE 28th International Parallel & Distributed Processing Symposium Workshops Constructing Similarity Graphs from Large-scale Biological Sequence Collections

Abstract—Detecting similar pairs in large biological sequence collections is one of the most commonly performed tasks in computational biology. With the advent of high throughput sequencing technologies the problem regained significance as data sets with millions of sequences became ubiquitous. This paper is an initial report on our parallel, distributed memory and sketching-based approach to constructing large-scale sequence similarity graphs. We develop load balancing techniques, derived from multi-way number partitioning and work stealing, to manage computational imbalance and ensure scalability on thousands of processors. Our experimental results show that the method is efficient, and can be used to analyze data sets with millions of DNA sequences in acceptable time limits. Keywords-sequence similarity; min-wise independent permutations; load balancing; parallel computational biology; I