A framework and a performance assessment for serverless MapReduce on AWS Lambda

[EN] MapReduce is one of the most widely used programming models for analysing large-scale datasets, i.e. Big Data. In recent years, serverless computing and, in particular, Functions as a Service (FaaS) has surged as an execution model in which no explicit management of servers (e.g. virtual machines) is performed by the user. Instead, the Cloud provider dynamically allocates resources to the function invocations and fine-grained billing is introduced depending on the execution time and allocated memory, as exemplified by AWS Lambda. In this article, a high-performant serverless architecture has been created to execute MapReduce jobs on AWS Lambda using Amazon S3 as the storage backend. In addition, a thorough assessment has been carried out to study the suitability of AWS Lambda as a platform for the execution of High Throughput Computing jobs. The results indicate that AWS Lambda provides a convenient computing platform for general-purpose applications that fit within the constraints of the service (15 min of maximum execution time, 3008 MB of RAM and 512 MB of disk space) but it exhibits an inhomogeneous performance behaviour that may jeopardise adoption for tightly coupled computing jobs.This study was supported by the program "Ayudas para la contratacion de personal investigador en formacion de caracter pre-doctoral, programa VALid-d" under grant number ACIF/2018/148 from the Conselleria d'Educacio of the Generalitat Valenciana, Spain. The authors would also like to thank the Spanish "Ministerio de Economia, Industria y Competitividad" for the project "BigCLOE" with reference number TIN2016-79951-R.Giménez-Alventosa, V.; Moltó, G.; Caballer Fernández, M. (2019). A framework and a performance assessment for serverless MapReduce on AWS Lambda. Future Generation Computer Systems. 97:259-274. https://doi.org/10.1016/j.future.2019.02.057S2592749