Collective Knowledge Playground

This presentation introduces Collective Knowledge Playground - a free, open-source and technology-agnostic on-prem platform being developed by the MLCommons taskforce on automation and reproducibility. Our goal is to let the community benchmark, optimize and compare AI, ML and other emerging applications across diverse and rapidly evolving models, software, hardware and data from different vendors in terms of costs, performance, power consumption, accuracy, size and other metrics in a unified, collaborative, automated and reproducible way. This platform is powered by the portable and technology-agnostic MLCommons Collective Mind automation framework (CM aka CK2) with portable and reusable automation recipes developed by the community to solve the "AI/ML dependency hell" and automatically connect diverse and continuously changing models, software, hardware, data sets, best practices and optimization techniques into end-to-end applications in a transparent and non-intrusive way. Our vision for the CK platform is to help researchers, engineers and entrepreneurs accelerate innovation by automatically generating the most efficient, reproducible and deployable full-stack AI/ML applications using the most suitable software/hardware stack at any given time (model, framework, inference engine and any other related dependency) based on their requirements and constraints including costs, throughput, latency, power consumption, accuracy, target devices (cloud/edge/mobile/tiny), environment and data while slashing their research, development and operational costs. See this ACM tech talk and journal article to learn more about our motivation