While Machine Learning is becoming more and more important in everyday life, designing efficient ML Systems and deploying them in the real world is becoming increasingly challenging, time consuming and costly. Researchers and engineers must keep pace with rapidly evolving software stacks and a Cambrian explosion of hardware platforms from the cloud to the edge. Such platforms have their own specific libraries, frameworks, APIs and specifications and often require repetitive, tedious and ad-hoc optimization of the whole model/software/hardware stack to trade off accuracy, latency, throughout, power consumption, size and costs depending on user requirements and constraints.
The Collective Knowledge framework (CK) is our attempt to develop a common plug&play infrastructure that can be used by the community similar to Wikipedia to learn how to solve above challenges and make it easier to co-design, benchmark, optimize and deploy Machine Learning Systems in the real world across continuously evolving software, hardware and data sets (see our ACM TechTalk for more details):
CK aims at providing a simple playground with minimal software dependencies to help researchers and practitioners share their knowledge in the form of reusable automation recipes with a unified Python API, CLI and meta description:
CK helps to organize software projects and Git repositories as a database of above automation recipes and related artifacts based on FAIR principles as described in our journal article (shorter pre-print). See examples of CK-compatible GitHub repositories:
We collaborated with the community to reproduce 150+ ML and Systems papers and implement the following reusable automation recipes in the CK format:
Portable meta package manager to automatically detect, install or rebuild various ML artifacts (ML models, data sets, frameworks, libraries, etc) across different platform and operating systems including Linux, Windows, MacOS and Android:
Portable manager for Python virtual environments: CK repo.
Portable workflows to support collaborative, reproducible and cross-platform benchmarking:
Portable workflows to automate MLPerf™ benchmark:
Further community developments are supported by MLCommons™, OctoML and the cTuning foundation within the MLCommons' Design Space Exploration workgroup. Please contact Grigori Fursin if you are interested to join this community effort!
The latest version of the CK automation suite supported by MLCommons™:
We plan to develop a new version of the CK framework (v3) within the MLCommons' Design Space Exploration workgroup - please contact Grigori Fursin to join this community effort!
Versions 1.x including v1.17.0 and 1.55.5 (BSD license) are not supported anymore. Please get in touch and we will help you to upgrade your infrastructure to use the latest MLCommons technology!
Follow this guide to install CK framework on your platform.
CK supports the following platforms:
|As a host platform||As a target platform|
|Bare-metal (edge devices)||-||±|
Here we show how to pull a GitHub repo in the CK format and use a unified CK interface to compile and run any program (image corner detection in our case) with any compatible data set on any compatible platform:
python3 -m pip install ck ck pull repo:mlcommons@ck-mlops ck ls program:*susan* ck search dataset --tags=jpeg ck pull repo:ctuning-datasets-min ck search dataset --tags=jpeg ck detect soft:compiler.gcc ck detect soft:compiler.llvm ck show env --tags=compiler ck compile program:image-corner-detection --speed ck run program:image-corner-detection --repeat=1 --env.MY_ENV=123 --env.TEST=xyz
You can check output of this program in the following directory:
cd `ck find program:image-corner-detection`/tmp ls processed-image.pgm
You can now view this image with detected corners.
Check CK docs for further details.
We have prepared adaptive CK containers to demonstrate MLOps capabilities:
You can run them as follows:
ck pull repo:mlcommons@ck-mlops ck build docker:ck-template-mlperf --tag=ubuntu-20.04 ck run docker:ck-template-mlperf --tag=ubuntu-20.04
You can create multiple virtual CK environments with templates to automatically install different CK packages and workflows, for example for MLPerf™ inference:
ck pull repo:mlcommons@ck-venv ck create venv:test --template=mlperf-inference-main ck ls venv ck activate venv:test ck pull repo:mlcommons@ck-mlops ck install package --ask --tags=dataset,coco,val,2017,full ck show env
All CK modules, automation actions and workflows are accessible as a micro-service with a unified JSON I/O API to make it easier to integrate them with web services and CI platforms as described here.
We have developed the cKnowledge.io portal to help the community organize and find all the CK workflows and components similar to PyPI:
The community provides Docker containers to test CK and components using different ML/SW/HW stacks (DSE).
Note, that we plan to redesign the CK core to be more pythonic (we wrote the first prototype without OO to be able to port it to bare-metal devices in C but eventually we decided to drop this idea).
Please consider joining the MLCommons' Design Space Exploration workgroup to join this community effort - contact Grigori Fursin for more details.