OV Operator 0.2 (openvinotoolkit#684)

added new version of the operator for openshift 0.2.0 


Co-authored-by: Rafal Sapala <[email protected]>

deploy/README.md

@@ -1,8 +1,9 @@
 # Kubernetes Deployment
 
-A helm chart for installing OpenVINO Model Server in a Kubernetes cluster is provided. By default, the cluster contains 
-a single instance of the server but the _replicas_ configuration parameter can be set to create a cluster 
-of any size, as described below. This guide assumes you already have a functional Kubernetes cluster and helm 
+A helm chart for installing OpenVINO Model Server in a Kubernetes cluster is provided. 
+The helm chart is managing the Model Server instance which represents a kubernetes deployment and a
+kubernetes service with exposed REST and gRPC inference endpoints.
+This guide assumes you already have a functional Kubernetes cluster and helm 
 installed (see below for instructions on installing helm).
 
 The steps below describe how to setup a model repository, use helm to launch the inference server and then send 
@@ -14,20 +15,27 @@ Please refer to [Helm installation guide](https://helm.sh/docs/intro/install).
 
 ## Model Repository
 
-If you already have a model repository you may use that with this helm chart. If you don't, you can use any model
-from [OpenVINO Model Zoo](https://download.01.org/opencv/2021/openvinotoolkit/2021.2/open_model_zoo/models_bin/).
+Model Server requires a repository of models to execute inference requests. That consists of the model files stored in a 
+specific structure. Each model is stored in a dedicated folder with numerical subfolders representing the model versions.
+Each model version subfolder must include its model files. 
 
-Model Server requires a repository of models to execute inference requests. For example, you can 
+Model repository can be hosted in the cloud storage, Kubernetes persistent volume or on the local drives.
+
+Learn more about the [model repository](../docs/models_repository.md)
+
+For example, you can 
 use a Google Cloud Storage (GCS) bucket:
 ```shell script
 gsutil mb gs://model-repository
 ```
 
-You can download the model from the link provided above and upload it to GCS:
+You can download the model from [OpenVINO Model Zoo](https://storage.openvinotoolkit.org/repositories/open_model_zoo/2021.3/models_bin). and upload it to GCS:
+
 ```shell script
-gsutil cp -r 1 gs://model-repository/1
+wget https://storage.openvinotoolkit.org/repositories/open_model_zoo/2021.3/models_bin/2/resnet50-binary-0001/FP32-INT1/resnet50-binary-0001.bin -P 1
+wget https://storage.openvinotoolkit.org/repositories/open_model_zoo/2021.3/models_bin/2/resnet50-binary-0001/FP32-INT1/resnet50-binary-0001.xml -P 1
+gsutil cp -r 1 resnet50-binary-0001.bin gs://model-repository/resnet
 ```
-The models repository can be also distributed on the cluster nodes in the local path or it could be stored on the Kubernetes persistent volume.
 
 The supported storage options are described below:
 
@@ -44,20 +52,20 @@ Bucket permissions can be set with the _GOOGLE_APPLICATION_CREDENTIALS_ environm
 
 * When deploying Model Server, provide the model path to GCS bucket and name for the secret created above. Make sure to provide `gcp_creds_secret_name` when deploying:
 ```shell script
-helm install ovms-app ovms --set model_name=resnet50-binary-0001,model_path=gs://models-repository,gcp_creds_secret_name=gcpcreds
+helm install ovms-app ovms --set model_name=resnet50-binary-0001,model_path=gs://models-repository/model,gcp_creds_secret_name=gcpcreds
 ```
 
 ### S3
 
 For S3 you must provide an AWS Access Key ID, the content of that key (AWS Secret Access Key) and the AWS region when deploying: `aws_access_key_id`, `aws_secret_access_key` and `aws_region` (see below)
 ```shell script
-helm install ovms-app ovms --set model_name=icnet-camvid-ava-0001,model_path=s3://models-repository,aws_access_key_id=<...>,aws_secret_access_key=<...>,aws_region=eu-central-1
+helm install ovms-app ovms --set model_name=icnet-camvid-ava-0001,model_path=s3://models-repository/model,aws_access_key_id=<...>,aws_secret_access_key=<...>,aws_region=eu-central-1
 ```
 
 In case you would like to use custom S3 service with compatible API (e.g. MinIO), you need to also provide endpoint 
 to that service. Please provide it by supplying `s3_compat_api_endpoint`:
 ```shell script
-helm install ovms-app ovms --set model_name=icnet-camvid-ava-0001,model_path=s3://models-repository,aws_access_key_id=<...>,aws_secret_access_key=<...>,s3_compat_api_endpoint=<...>
+helm install ovms-app ovms --set model_name=icnet-camvid-ava-0001,model_path=s3://models-repository/model,aws_access_key_id=<...>,aws_secret_access_key=<...>,s3_compat_api_endpoint=<...>
 ```
 
 ### Azure Storage
@@ -131,7 +139,7 @@ Deploy Model Server using _helm_. Please include the required model name and mod
 helm install ovms-app ovms --set model_name=resnet50-binary-0001,model_path=gs://models-repository
 ```
 
-Use _kubectl_ to see status and wait until the Model Server pod is running:
+Use _kubectl_ to see the status and wait until the Model Server pod is running:
 ```shell script
 kubectl get pods
 NAME                    READY   STATUS    RESTARTS   AGE
@@ -182,6 +190,10 @@ ovms-app   LoadBalancer   10.121.14.253   1.2.3.4         8080:30043/TCP,8081:32
 
 The server exposes an gRPC endpoint on 8080 port and REST endpoint on 8081 port.
 
+The service name deployed via the helm chart is defined by the application name. In addition to that, the service
+gets a suffix `-ovms`, in case the application name doesn't include `ovms` phrase. It avoids a risk of the service name
+conflicts with other application.
+
 Follow the [instructions](https://github.com/openvinotoolkit/model_server/tree/main/example_client#submitting-grpc-requests-based-on-a-dataset-from-a-list-of-jpeg-files) 
 to create an image classification client that can be used to perform inference with models being exposed by the server. For example:
 ```shell script
@@ -246,10 +258,16 @@ release "ovms-app" uninstalled
 | config_path | Starts OVMS using the config file mounted from the node local path or the k8s persistent volume | Use it together with models_host_path or models_claim_name and place the config file in configured storage path | - |
 | grpc_port      | service port for gRPC interface |  | 8080 |
 | grpc_port      | service port for REST API interface |  | 8081 |
+| file_system_poll_wait_seconds      | Time interval in seconds between new version detection. 0 disables the version updates |  | 1 |
 | model_name      | model name, start OVMS with a single model, excluding with config_configmap_name and config_path parameter |  | - |
 | model_path      | model path, start OVMS with a single model, excluding with config_configmap_name and config_path parameter |  | - |
 | target_device      | Target device to run inference operations | Non CPU device require the device plugin to be deployed | CPU |
+| stateful      | If set to any non empty value, enables stateful model execution | Model must be stateful | Stateless model execution |
 | nireq      | Size of inference queue  |  | set automatically by OpenVINO|
+| batch_size      | Change the model batch size  |  | defined by the model attributes |
+| layout      | Change layout of the model input or output with image data. NCHW or NHWC  |  | Defined in the model |
+| shape      | Change the model input shape  |  | defined by the model attributes |
+| model_version_policy      | Set the model version policy  |  | {\"latest\": { \"num_versions\":1 }} The latest version is served |
 | plugin_config      | Device plugin configuration used for performance tuning  |  | {\"CPU_THROUGHPUT_STREAMS\":\"CPU_THROUGHPUT_AUTO\"}|
 | gcp_creds_secret_name      | k8s secret resource including GCP credentials, use it with google storage for models | Secret should be created with GCP credentials json file | - |
 | aws_access_key_id      | S3 storage access key id, use it with S3 storage for models |  | - |
@@ -261,6 +279,8 @@ release "ovms-app" uninstalled
 | log_level      | OVMS log level, one of ERROR,INFO,DEBUG|  |  INFO |
 | service_type      | k8s service type |  | LoadBalancer |
 | resources      | compute resource limits |  | All CPU and memory on the node |
+| node_selector      | target node label condition |  | All available nodes |
+| annotations      | defined annotations to be set in the pods |  | none |
 | security_context     | OVMS security context |  | 5000:5000 |
 | models_host_path      | mounts node local path in container as /models folder | Path should be created on all nodes and populated with the data | - |
 | models_volume_claim      | mounts k8s persistent volume claim in the container as /models | Persistent Volume Claim should be create in the same namespace and populated with the data | - |

deploy/ovms/Chart.yaml

@@ -17,4 +17,4 @@
 apiVersion: v1
 description: OpenVINO Model Server
 name: ovms
-version: 3.0.0
+version: 4.0.0

deploy/ovms/templates/deployment.yaml

@@ -32,6 +32,8 @@ spec:
   replicas: {{ .Values.replicas }}
   template:
     metadata:
+      annotations:
+{{ toYaml .Values.annotations | indent 8 }}
       labels:
         heritage: {{ .Release.Service | quote }}
         release: {{ .Release.Name | quote }}
@@ -115,8 +117,26 @@ spec:
                {{- if .Values.plugin_config }}
                "--plugin_config", "{{ .Values.plugin_config }}",
                {{- end }}
+               {{- if .Values.batch_size }}
+               "--batch_size", "{{ .Values.batch_size }}",
+               {{- end }}
+               {{- if .Values.layout }}
+                "--layout", "{{ .Values.layout }}",
+               {{- end }}
+               {{- if .Values.shape }}
+               "--shape", "{{ .Values.shape }}",
+               {{- end }}
+               {{- if .Values.model_version_policy }}
+               "--model_version_policy", "{{ .Values.model_version_policy }}",
+               {{- end }}
+               {{- if .Values.stateful }}
+               "--stateful",
+               {{- end }}
         {{- end }}
                "--log_level", "{{ .Values.log_level }}",
+               {{- if .Values.file_system_poll_wait_seconds }}
+               "--file_system_poll_wait_seconds", "{{ .Values.file_system_poll_wait_seconds }}",
+               {{- end }}
                "--port", "8080",
                "--rest_port", "8081"]
         {{- if or .Values.gcp_creds_secret_name .Values.config_configmap_name .Values.models_host_path .Values.models_volume_claim}}
@@ -139,6 +159,10 @@ spec:
         {{- end }}
         resources:
 {{ toYaml .Values.resources | indent 10}}
+      {{- if .Values.node_selector }}
+      nodeSelector:
+{{ toYaml .Values.node_selector | indent 8 }}
+      {{- end }}
       {{- if or .Values.gcp_creds_secret_name .Values.config_configmap_name .Values.models_volume_claim .Values.models_host_path }}
       volumes:
       {{- end }}

deploy/ovms/values.yaml

@@ -22,13 +22,19 @@ config_path: ""
 # to be used when ovms config is not included in the configmap but is mounted on the container via models_host_path or models_volume_claim param
 grpc_port: 8080
 rest_port: 8081
+file_system_poll_wait_seconds: 0
 model_name: ""
 model_path: ""
+layout: ""
 target_device: "CPU"
 nireq: ""
+batch_size: ""
+shape: ""
+model_version_policy: >-
+  {\"latest\": { \"num_versions\":1 }}
 plugin_config: >-
   {\"CPU_THROUGHPUT_STREAMS\":\"CPU_THROUGHPUT_AUTO\"}
-
+stateful: ""
 gcp_creds_secret_name: ""
 # k8s secret name holding gcp credential file gcp-creds.json
 aws_access_key_id: ""
@@ -47,6 +53,10 @@ resources: {}
 security_context: {}
   #  runAsUser: 5000
   #  runAsGroup: 5000
+annotations: {}
+  # sidecar.istio.io/inject: "true"
+node_selector: {}
+  # kubernetes.io/arch: amd64
 models_host_path: ""
 # local path on the host to be mounted as /models directory
 models_volume_claim: ""

extras/openvino-operator-openshift/.gitignore

@@ -0,0 +1,14 @@
+
+# Binaries for programs and plugins
+*.exe
+*.exe~
+*.dll
+*.so
+*.dylib
+bin
+
+# editor and IDE paraphernalia
+.idea
+*.swp
+*.swo
+*~

extras/openvino-operator-openshift/Dockerfile

@@ -0,0 +1,24 @@
+#
+# Copyright (c) 2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM quay.io/operator-framework/helm-operator:v1.7.2
+LABEL "Description"="OpenVINO(TM) Model Server Operator"
+LABEL "Vendor"="Intel Corporation"
+ENV HOME=/opt/helm
+COPY watches.yaml ${HOME}/watches.yaml
+COPY helm-charts  ${HOME}/helm-charts
+COPY LICENSE LICENSE
+WORKDIR ${HOME}