Optimizations for distributed data analytics in cloud environment

M.Phil.Cloud computing has become more and more important in our daily lives. Efficiently utilizing the resources on cloud servers becomes the major focus of cloud computing service providers. In this thesis, we study how we can optimize distributed data analytics workloads in the cloud environment by better utilizing networking resources.Firstly, we discuss how we can reduce cross-data-center bandwidth usage of a data analytics platform of Alibaba deployed on geographically distributed data centers. We decompose the problem into multiple sub-problems and devised algorithms to efficiently solve these problems respectively. The new solutions we proposed help to reduce the cross-data-center bandwidth usage by tens of PBs per day.Secondly, we design an algorithm which enables distributed deep neural network training workloads to adjust to dynamic network conditions in the cloud environment. The algorithm detects the network condition changes through the changes of training throughput, and helps the distributed DNN training workloads to adjust to changes by better utilizing the current share of network bandwidth.雲計算成為了我們生活中越來越重要的一部分。如何更有效地使用雲服務器上的計算資源成為了雲服務提供商所重點關注的問題。在這篇論文中，我們將學習如何更好地利用雲上的網絡資源來提升雲上作業的性能。首先，我們討論如何減少阿里巴巴公司一個部署在跨地域數據中心上的分佈式大數據分析平台的跨數據中心網絡帶寬用量。我們將這個問題分解成為多個子問題，并分別設計有效的算法去解決這些問題。我們提出的這些方法可以幫助我們每天減少數十PB的跨數據中心網絡帶寬用量。接著，我們設計了一個可以讓分佈式神經網絡訓練作業適應雲環境下動態網絡環境的算法。這個算法利用訓練吞吐量的變化來檢測網絡環境變化，並且使得分佈式神經網絡訓練作業能夠適應變化，從而更好地利用當前狀況下的網絡帶寬。Zhong, Zheng.Thesis M.Phil. Chinese University of Hong Kong 2020.Includes bibliographical references (leaves 57-65).Abstracts also in Chinese.Title from PDF title page (viewed on 7, October 2021)