Cloud Computing Infrastructure for Data Intensive Applications

This chapter describes the general architecture and functional components of the cloud-based big data infrastructure (BDI). The chapter starts with the analysis of emerging Big Data and data intensive technologies and provides the general definition of the Big Data Architecture Framework (BDAF) that includes the following components: Big Data definition, Data Management including data lifecycle and data structures, generically cloud based BDI, Data Analytics technologies and platforms, and Big Data security, compliance, and privacy. The chapter refers to NIST Big Data Reference Architecture (BDRA) and summarizes general requirements to Big Data systems described in NIST documents. The proposed BDI and its cloud-based components are defined in accordance with the NIST BDRA and BDAF. This chapter provides detailed analysis of the two bioinformatics use cases as typical example of the Big Data applications that have being developed by the authors in the framework of the CYCLONE project. The effective use of cloud for bioinformatics applications requires maximum automation of the applications deployment and management that may include resources from multiple clouds and providers. The proposed CYCLONE platform for multicloud multiprovider applications deployment and management is based on the SlipStream cloud automation platform and includes all necessary components to build and operate complex scientific applications. The chapter discusses existing platforms for cloud powered applications development and deployment automation, in particularly referring to the SlipStream cloud automation platform, which allows multicloud applications deployment and management. The chapter also includes a short overview of the existing Big Data platforms and services provided by the major cloud services providers which can be used for fast deployment of customer Big Data applications using the benefits of cloud technologies and global cloud infrastructure