OpenVINO™ Model Server

OpenVINO™ Model Server (OVMS) is a scalable, high-performance solution for serving machine learning models optimized for Intel® architectures. The server provides an inference service via gRPC or REST API - making it easy to deploy new algorithms and AI experiments using the same architecture as TensorFlow Serving for any models trained in a framework that is supported by OpenVINO.

The server implements gRPC and REST API framework with data serialization and deserialization using TensorFlow Serving API, and OpenVINO™ as the inference execution provider. Model repositories may reside on a locally accessible file system (e.g. NFS), Google Cloud Storage (GCS), Amazon S3, Minio or Azure Blob Storage.

OVMS is now implemented in C++ and provides much higher scalability compared to its predecessor in Python version. You can take advantage of all the power of Xeon CPU capabilities or AI accelerators and expose it over the network interface. Read release notes to find out what's new in C++ version.

Review the Architecture concept document for more details.

A few key features:

Support for multiple frameworks. Serve models trained in popular formats such as Caffe*, TensorFlow*, MXNet* and ONNX*.
Online deployment of new model versions.
Configuration updates in a runtime
Support for AI accelerators including Intel Movidius Myriad VPUs, GPU and HDDL.
The server can be enabled both on Bare Metal Hosts or in Docker containers.
Model reshaping. The server supports reshaping models in runtime.
Directed Acyclic Graph Scheduler Connect multiple models to deploy complex processing solutions and reduce overhead of sending data back and forth.
Custom nodes in DAG pipelines Model inference or data transformations can be implemented by a custom node C/C++ implementation loaded as an external library.
Serving stateful models. Serve models that operate on sequences of data and maintain state between inference requests.
Binary format of the input data. Input data can be sent in JPEG or PNG format to reduce traffic and offload the client applications.

Note: OVMS has been tested on RedHat*, CentOS* and Ubuntu*. Latest publicly released docker images are based on Ubuntu and UBI. They are stored in:

Run OpenVINO Model Server

A demonstration how to use OpenVINO Model Server can be found in a quick start guide.

More detailed guides to using Model Server in various scenarios can be found here:

API documentation

GRPC

Learn more about GRPC API

Refer to the GRPC example client code to learn how to use and submit the requests using the gRPC interface.

REST

Learn more about REST API

Refer to the REST API example client code to learn how to use REST API

OVMS Python Client Library

For simplified interaction with the model server API, the Python client library has been introduced. It's a set of Python functions and objects that wrap things like:

setting connection with the server
creating TensorProto from data
creating requests for model status, model metadata and prediction
sending requests to appropriate endpoints

Testing

Learn more about tests in the developer guide

Known Limitations

Currently, Predict, GetModelMetadata and GetModelStatus calls are implemented using Tensorflow Serving API.
Classify, Regress and MultiInference are not included.
Output_filter is not effective in the Predict call. All outputs defined in the model are returned to the clients.