Integration of Apache Mesos over GPUs resources in the DEEP Hybrid DataCloud project

Trabajo presentado a IBERGRID: Delivering Innovative Computing and Data services to Researchers, celebrado en Santiago de Compostela (España) del 23 al 26 de septiembre de 2019.DEEP Hybrid DataCloud project was proposed with the necessity to support different amount of intensive computing techniques over specialized hardware, like HPC or GPUs. The project focus on the integration of this specialized, and expensive, hardware under a Cloud Platform as OpenStack that can be used on-demand by researchers of different areas. Within this project, a set of building blocks whose solution is called >DEEP as a Service> was implemented to make the application deployment more easier for the user. For this development, it is necessary to provide the researchers with access to these technologies as friendly but powerful services able to exploit very large datasets. On the one hand we have the gpu resources and on the other hand the users and their applications to run over those resources. DEEP needs to provide a service that controls how users can use those resources in an efficiently way. Although there are multiple technologies that address this problem as a queuing system, Apache Mesos has been developed to do it in an effective and controlled manner. Apache Mesos is a technology that abstracts the resources from different machines, like cpu, gpu, ram and storage to provide a scheduling and distributed system across the whole cloud environment. Mesos is easily to deploy over cpu-based systems but gpu needs a more tricky configuration as it is shown in this solution. As an added value, this solution provides an apache2 configuration for authenticate users from different communities by the current AAI service in DEEP. The proposed presentation will show the design and deployment of Mesos to work over gpus resources inside the Deep Hybrid DataCloud Project scope.This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 777435