Machine Learning Model Management using Cloud-Native Technologies for IoT

Thanks to many breakthroughs in neural network techniques, machine learning is widely applied in many applications, including IoT. However, the challenge is to run a machine learning pipeline in production without introducing a huge technical debt. Edge computing and IoT have become popular thanks to the advancements in networking and virtualization. Cloud-native technologies, in general, and Kubernetes, in specific, can manage a large scale cluster to run a massive fleet of containers. Thus, running machine learning pipelines on Kubernetes for IoT is a great match. However, any abstraction layer would introduce overhead. The question is how large the overhead is to run a machine learning model workflow on Kubernetes. Also, cloud-native technology enables easy management of Edge computing resources. The question is how efficient it is to update machine learning models used by a serving server at an Edge node. To address the first research question, we measure the execution time overhead of Kubernetes jobs compiled from Jupyter notebook running an E2E machine learning application. The experiment is conducted with different dataset sizes and machine learning model sizes to investigate how they affect the overhead. To address the second research question, we measure and compare the machine model uploading time to an object-storage service in two setups: deployed as a Kubernetes service and a service running in a virtual machine. The measurements are conducted with artificial network latency to simulate serving machine learning model at the Edge. In our experiments, we observe the overhead introduced by the Kubernetes and the pipeline applications; however, the overhead becomes less significant as working with bigger datasets. Because of the expensive model training job, the overall overhead is much less comparing the training job computation time. In the context of update machine learning models at the Edge node, to upload the model to an object-storage Kubernetes service shows no significant difference to another service serving at a virtual machine without Kubernetes. On the other hand, the experiments also show that cloud-native technologies offer excellent abstractions and automation to manage cloud resources for containerization