mlserver

An inference server for your machine learning models, including support for multiple frameworks, multi-model serving and more

Showing:

Popularity

Downloads/wk

0

GitHub Stars

78

Maintenance

Last Commit

4d ago

Contributors

10

Package

Dependencies

8

License

Apache 2.0

Categories

Readme

MLServer

An open source inference server for your machine learning models.

Overview

MLServer aims to provide an easy way to start serving your machine learning models through a REST and gRPC interface, fully compliant with KFServing's V2 Dataplane spec.

You can read more about the goals of this project on the inital design document.

Usage

You can install the mlserver package running:

pip install mlserver

Note that to use any of the optional inference runtimes, you'll need to install the relevant package. For example, to serve a scikit-learn model, you would need to install the mlserver-sklearn package:

pip install mlserver-sklearn

For further information on how to use MLServer, you can check any of the available examples.

Inference Runtimes

Inference runtimes allow you to define how your model should be used within MLServer. You can think of them as the backend glue between MLServer and your machine learning framework of choice. You can read more about inference runtimes in their documentation page.

Out of the box, MLServer comes with a set of pre-packaged runtimes which let you interact with a subset of common frameworks. This allows you to start serving models saved in these frameworks straight away.

Out of the box, MLServer provides support for:

FrameworkSupportedDocumentation
Scikit-Learn👍MLServer SKLearn
XGBoost👍MLServer XGBoost
Spark MLlib👍MLServer MLlib
LightGBM👍MLServer LightGBM
Tempo👍github.com/SeldonIO/tempo
MLflow👍MLServer MLflow

Examples

To see MLServer in action, check out our full list of examples. You can find below a few selected examples showcasing how you can leverage MLServer to start serving your machine learning models.

Developer Guide

Versioning

Both the main mlserver package and the inference runtimes packages try to follow the same versioning schema. To bump the version across all of them, you can use the ./hack/update-version.sh script. For example:

./hack/update-version.sh 0.2.0.dev1

Rate & Review

Great Documentation0
Easy to Use0
Performant0
Highly Customizable0
Bleeding Edge0
Responsive Maintainers0
Poor Documentation0
Hard to Use0
Slow0
Buggy0
Abandoned0
Unwelcoming Community0
100