1

Many scientific problems depend on the ability to analyze and compute on large amounts of data. This analysis often does not scale well, i.e. its effectiveness is hampered by the increasing volume, variety and rate of change (velocity) of big data. There is a need to integrate features of traditional high-performance computing, such as scientific libraries, communication and resource management middleware, with the rich set of capabilities found in the commercial Big Data ecosystem, resulting in an integrated system generi-cally called high-performance big data system (HPBDS). Our proposed preliminary implementation of the HPBDS – includes many important software systems such as Hadoop available from the Apache open source community and thus referred to as High-Performance Computing-Big Data Stack (HPC–ABDS) – has two fundamental building blocks: (i) Middleware for Data-Intensive Analytics and Science (MIDAS) that will enable scalable applications with the performance of HPC (High Performance Computing) and the rich functionality of the commodity Apache Big Data Stack. (ii) The second building block will design and imple-ment a set of cross-cutting high-performance data-analysis libraries SPIDAL (Scalable Parallel Interopera-ble Data Analytics Library), which will support new programming and execution models for data-intensive analysis in a wide range of science and engineering applications. These libraries will be implemented to be scalable and interoperable across a range of computing systems including clouds, clusters and supercomput